Hi, my name is Nicole Vicinanza and I’m a Senior Research Associate with JBS International, a consulting firm. In my consulting work, one of my roles is to provide technical assistance in evaluation to community based organizations, government programs and service providers whose primary job is not evaluation.
Hot Tip: How do you explain random sampling to folks for whom sampling is a new requirement? Try using candy to show your clients the impact that different approaches and sample sizes can have. I’ve done this with groups by using different colored Hershey’s kisses in paper bags, but any similarly shaped, but different colored items will do. It allows folks to see the impact of different samples in concrete ways, cheaply, quickly and edibly.
First set up the bags with one color (e.g. silver) representing the most common type of respondent, and then other colors (e.g. purple and silver and tan/caramel) in much smaller numbers to represent respondents who have specific issues or problems. Set up the bags so that the total number of “respondents” and “problems” is easy to remember, or write down what you’ve put in the bags. Introduce the issue of sampling to the group and hand out the bags (either to individuals or table groups), but don’t tell them the proportion or type of “problems” that are in their bag. Use the bags to try out different approaches to sampling and sample sizes. Folks can look, and then quickly pick what they think is a representative sample, draw “blind” samples of different sizes, or pull larger or smaller samples from different bags. After they’ve pulled their first sample, tell them what the different colors represent, then discuss how that knowledge might change the size of the sample they pull. Try different random sample sizes, and record the results you get on worksheets or flip charts. Once you’ve finished trying samples of different sizes, tell them the proportions of different “problems” in their bags. How close did the different sample sizes come to the actual proportions? Discuss which approaches and sample sizes worked best for different information needs (e.g. uncovering different types problems vs. estimating proportions of problems). Talk about how moving from candies to sampling with real people may impact the results- which can move you into a discussion about non-response bias, how people with different issues may respond differently to you data collection, and cost issues.
Note: If folks can see what they’re pulling, they may bias the random samples- try having one person hold the bag, and another pull with their eyes shut. Also, don’t start eating until you’ve pulled the last sample- otherwise your pre-set proportions will get thrown off!
The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.