Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hello! We are Kim Firth Leonard, Leonard Research & Evaluation, LLC, and Oregon Community Foundation, and Sheila B. Robinson, Custom Professional Learning, LLC, co-authors of the text Designing Quality Survey Questions (Sage, 2018). We met on Twitter and now have been writing about survey design together for 10 years via our blogs, working with clients to design surveys, teaching survey design workshops, and presenting about survey design at conferences, including AEA.
Through this collaboration, we’ve learned and shared a lot of advice about how to maximize survey effectiveness – getting rich, nuanced, meaningful, and useful data – through respondent-centered design. Here are a few of our favorite interrelated “hot tips” from this work:
Hot Tip #1: Practice with ‘bad questions’ to build skill in spotting survey design flaws. A favorite activity during our workshops is reviewing bad survey questions. We do this not to poke fun at the poor folks who wrote the questions but to hone our ability to spot problems and fix them (OK, full disclosure: we do have a little giggle with our participants about bad survey questions recognizing that we’ve made some of those same mistakes ourselves. A little humor is good for learning!) It is so easy to make seemingly simple mistakes. In fact some of our “best” examples of bad questions come from multi-billion dollar corporations and huge government agencies. We find that the more we practice identifying those problems and addressing them, the better we get at avoiding them altogether.
Hot Tip #2: Take the time to know and understand your desired respondents as much as possible. The more we understand the perspectives, expectations, needs, and strengths of our desired respondents, the more likely we are to write questions that are meaningful and engaging as well as understandable and answerable! When we work with private clients or create surveys for evaluation projects we start by asking lots of questions about respondents. Then, we work through a pre-testing process that can be relatively informal or formal. Pre-testing can range from having a few potential respondents review a survey draft and provide feedback to a full formal survey pilot study. (Readers: You just got a bonus Hot Tip: pre-test everything!)
Hot Tip #3: Remember that the right way to write a survey question always depends on context. There is no single correct way to write any given survey question. What language to use in the question stem, what response options to offer, what question type is best, and other factors depend on what you are attempting to learn and measure, how you will analyze, interpret, and use the data, and most importantly, what respondents can answer accurately without too much mental effort. Start with your core research or evaluation questions, articulate a clear purpose for conducting the survey, and ensure that all survey questions are aligned with those. Consider context at every point during the survey design process.
Rad Resources: To learn more about survey design, we invite you to visit and subscribe to our blogs (Sheila, Kim), find a copy of our book (BONUS: SAGE offers two free sample chapters!), and connect with us (we’re both on LinkedIn) to be the first to know about upcoming public webinars and workshops. We also love to hear from fellow evaluators about their own survey conundrums and lessons learned – so please reach out!
We have a long list of favorite resources on survey design but one we love sharing is Asking Questions About Behavior: Cognition, Communication and Questionnaire Construction.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Nice hot tips! Tip 3 is probably the most important since it talks about data. All too often people create survey questions that can not be properly analyzed. Try not to solely collect qualitative data. This refers to answers like the following: VERY SATISFIED, NOT SATISFIED. You can still use those descriptors but overlay them with a number scale starting at 0. For example: 0 = very dissatisfied and 5 = very satisfied. Now your results start with a 0 position and the intervals between options have magnitude. In the end, you can create reports that provide salient conclusions and ideas for future change and business growth. Once again, thanks for the hot tips Elizabeth.