Susan Kistler on Survey Response Rate, Sample Size, Disclosure, from AAPOR

My name is Susan Kistler and I am AEA’s Executive Director. I contribute each Saturday’s aea365 post. Many years ago I taught survey and instrument design at the University of Minnesota, and survey methodology is an issue near and dear to my heart. Thus, this week, I want to share a few resources from the American Association of Public Opinion Research (AAPOR), an organization dedicated to public opinion and survey research. Their website is a good place to start if you have a question about developing and deploying an evaluation survey.

Rad Resource: AAPOR’s Standard Definitions report details how to calculate response rates for random dial telephone surveys, in-person household surveys, mail surveys, and internet surveys. It is essential reading for those conducting rigorous surveys.

Lessons Learned: AAPOR’s Standard Definitions provides a common language for talking about survey response rate, including differentiating among the following:

• Response rates – The number of complete interviews with reporting units divided by the number of eligible reporting units in the sample. The report provides six definitions of response rates, ranging from the definition that yields the lowest rate to the definition that yields the highest rate, depending on how partial interviews are considered and how cases of unknown eligibility are handled.
• Cooperation rates – The proportion of all cases interviewed of all eligible units ever contacted. The report provides four definitions of cooperation rates, ranging from a minimum or lowest rate, to a maximum or highest rate.
• Refusal rates – The proportion of all cases in which a housing unit or the respondent refuses to be interviewed, or breaks-off an interview, of all potentially eligible cases. The report provides three definitions of refusal rates, which differ in the way they treat dispositions of cases of unknown eligibility.
• Contact rates – The proportion of all cases in which some responsible housing unit member was reached. The report provides three definitions of contact rates.
  (page 4-5)

Rad Resource: AAPOR’s Response Rate Calculator is an Excel spreadsheet with the categories pre-identified from the Standard Definitions and with the needed formulas entered. It simplifies the job of translating the Standard Definitions from paper to practice.

Rad Resource: AAPOR’s Survey Disclosure Checklist details the minimum disclosure requirements when reporting survey results to the public.

Rad Resource: AAPOR recommends the Survey Random Sample Size Calculator from custominsight.com and includes details on good samples versus bad samples, margin of sampling error, and weighting on its What is a Random Sample page.

Thank you to AAPOR for so generously sharing publicly these – and so many other – resources.

The above represents my own opinions and not necessarily those of AEA.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.