We are Linda Cabral and Judy Savageau from the University of Massachusetts Medical School’s Center for Health Policy and Research. All of our projects involve some type of data collection, sometimes in the form of a survey. In order to get high quality survey data, you need to ensure that your respondents are interpreting questions in the way you intended. The familiarity and meaning of words may not be the same among all members of your sample. To increase the likelihood of high quality data, most of our evaluation protocols involving surveys include cognitive interviewing (aka ‘think aloud interviewing’ or ‘verbal probing’) as part of the survey design and pretesting process.
Cognitive interviewing, a qualitative approach to collecting quantitative data, enables evaluators to explore the processes by which respondents answer questions and the factors which influence their answers. For surveys, it involves fielding an instrument with a small group of individuals from your target sample population and asking the following types of questions for each item:
- Are you able to answer this question? If not, why not?
- Is this question clear? If not, what suggestions do you have for making it clearer?
- How do you interpret this question? Or, how do you interpret specific words or phrases within a question?
- Do the response options make sense? If not, what suggestions do you have?
- How comfortable are you answering this question?
Cognitive interviewing can reduce respondent burden by removing ambiguity and adding clarity so that when the survey is launched, respondents will have an easier time completing it and give you the information needed for your evaluation.
- This technique will likely be new for respondents; their inclination will be to answer the survey question rather than talk about how they think about the question. Some up-front coaching will probably be needed, especially if you’re developing a survey for non-English speaking respondents.
- Cognitive interviewing can be a time consuming activity (and, thus, costly). Consider whether there are certain survey questions that will benefit more than others; e.g., undertaking this testing for simple demographic questions is likely unnecessary.
- A comprehensive pretesting process includes both cognitive interviewing and pilot testing of the instrument. Whereas the primary goal of cognitive testing is to identify how questions are interpreted and revise questions as needed, pilot testing extends this process by examining length, flow, salience, and ease of the survey’s administration. Pilot testing may detect more concrete problems with the survey overall that may affect responses to specific questions and/or the overall response rate.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.