AEA365 | A Tip-a-Day by and for Evaluators

Apr/15

12

Pei-Pei Lei and Carla Hillerns on Smart Survey Design: Randomization of Response Options

Hi, we are Pei-Pei Lei and Carla Hillerns from the Office of Survey Research at the University of Massachusetts Medical School. Have you ever selected an answer to a survey question without reading all of the choices? Maybe you paid more attention to the first choice than the rest? Today, we’d like to share a technique that helps to minimize the impact of these types of scenarios – randomized ordering of survey response options.

Randomizing the order of response options may improve data quality by reducing the order effect in your survey. When there is a list of response options, respondents often have a tendency of selecting the most prominent. For example, in a paper survey, the first option may be most apparent. In a phone survey, the last option may be most memorable. If implementing an online survey, there may be a tendency to choose from the middle of a long list – because the center is more prominent.

By randomizing the order, all options have the same possibility of appearing in each response position. In Example A below, “Direct mail” appears in the top spot. However, in Example B, the responses have been randomly reassigned and “Television” now appears at the top.

Lei

Hot Tips:

  • Do not randomize the order if the response options are better suited to a pre-determined sequence, such as months of the year or alphabetization, or if using a validated instrument that needs to maintain the full survey as developed.
  • If the response list is divided into sub-categories, you can randomize the category order as well as the items within each category.
  • If your list includes “Other (Please specify: __________)” or “None of the above”, keep these at the bottom so the question makes sense!
  • If using the same set of response options for multiple questions, apply the first randomized ordering to the subsequent questions to avoid confusion.
  • Randomization is not a cure for all questionnaire design challenges. For example, respondents probably won’t pay as much attention to each response option if the list is extremely long or the options are excessively wordy. So be reasonable in your design.

Lesson Learned: It’s easy to administer randomization in web and telephone surveys if your survey platform supports this function. A mail survey will require multiple versions of the questionnaire. You’ll also need to account for these multiple versions as part of the data entry process to ensure that responses are coded accurately. 

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

2 comments

  • Elizabeth Brown · April 12, 2015 at 9:48 pm

    So the idea of randomization is not a “new” thing, but with the whole online-based survey, it definitely is a nifty trick/tool to utilize! I have to admit though, I have not really pushed the randomization of choices with others, but will definitely make sure to talk about the importance of using this approach (when appropriate). As for the Encyclopedia of Survey of Research Methods, it looks like a great resource! I plan on doing more research on it before I ask the powers that be to purchase it for us 🙂

    Reply

    • Pei-Pei Lei · April 30, 2015 at 2:36 pm

      Hi Elizabeth,
      Thank you for your feedback! Yes please think about using the randomization function if you have the right questionnaire and the easy-to-use tools (online survey programmer). Encyclopedia of Survey of Research Methods has covered a wide-range of topics we might encounter in the survey world. We got access through our University.

      Thanks,
      Pei-Pei and Carla

      Reply

Leave a Reply

<<

>>

Archives

To top