Pei-Pei Lei and Carla Hillerns on Smart Survey Design: Randomization of Response Options

Hi, we are Pei-Pei Lei and Carla Hillerns from the Office of Survey Research at the University of Massachusetts Medical School. Have you ever selected an answer to a survey question without reading all of the choices? Maybe you paid more attention to the first choice than the rest? Today, we’d like to share a technique that helps to minimize the impact of these types of scenarios – randomized ordering of survey response options.

Randomizing the order of response options may improve data quality by reducing the order effect in your survey. When there is a list of response options, respondents often have a tendency of selecting the most prominent. For example, in a paper survey, the first option may be most apparent. In a phone survey, the last option may be most memorable. If implementing an online survey, there may be a tendency to choose from the middle of a long list – because the center is more prominent.

By randomizing the order, all options have the same possibility of appearing in each response position. In Example A below, “Direct mail” appears in the top spot. However, in Example B, the responses have been randomly reassigned and “Television” now appears at the top.

Hot Tips:

• Do not randomize the order if the response options are better suited to a pre-determined sequence, such as months of the year or alphabetization, or if using a validated instrument that needs to maintain the full survey as developed.
• If the response list is divided into sub-categories, you can randomize the category order as well as the items within each category.
• If your list includes “Other (Please specify: __________)” or “None of the above”, keep these at the bottom so the question makes sense!
• If using the same set of response options for multiple questions, apply the first randomized ordering to the subsequent questions to avoid confusion.
• Randomization is not a cure for all questionnaire design challenges. For example, respondents probably won’t pay as much attention to each response option if the list is extremely long or the options are excessively wordy. So be reasonable in your design.

Lesson Learned: It’s easy to administer randomization in web and telephone surveys if your survey platform supports this function. A mail survey will require multiple versions of the questionnaire. You’ll also need to account for these multiple versions as part of the data entry process to ensure that responses are coded accurately. 

Rad Resources:

• The Encyclopedia of Survey Research Methods offers more information on response order effects.
• Check with your survey vendor or survey platform on the functionality that’s available to you.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.