AEA365 | A Tip-a-Day by and for Evaluators

Feb/10

23

Nicole Vicinanza on Explaining Random Sampling to Stakeholders

Hi, my name is Nicole Vicinanza and I’m a Senior Research Associate with JBS International, a consulting firm. In my consulting work, one of my roles is to provide technical assistance in evaluation to community based organizations, government programs and service providers whose primary job is not evaluation.

Hot Tip: How do you explain random sampling to folks for whom sampling is a new requirement? Try using candy to show your clients the impact that different approaches and sample sizes can have. I’ve done this with groups by using different colored Hershey’s kisses in paper bags, but any similarly shaped, but different colored items will do. It allows folks to see the impact of different samples in concrete ways, cheaply, quickly and edibly.

First set up the bags with one color (e.g. silver) representing the most common type of respondent, and then other colors (e.g. purple and silver and tan/caramel) in much smaller numbers to represent respondents who have specific issues or problems. Set up the bags so that the total number of “respondents” and “problems” is easy to remember, or write down what you’ve put in the bags. Introduce the issue of sampling to the group and hand out the bags (either to individuals or table groups), but don’t tell them the proportion or type of “problems” that are in their bag. Use the bags to try out different approaches to sampling and sample sizes. Folks can look, and then quickly pick what they think is a representative sample, draw “blind” samples of different sizes, or pull larger or smaller samples from different bags. After they’ve pulled their first sample, tell them what the different colors represent, then discuss how that knowledge might change the size of the sample they pull. Try different random sample sizes, and record the results you get on worksheets or flip charts. Once you’ve finished trying samples of different sizes, tell them the proportions of different “problems” in their bags. How close did the different sample sizes come to the actual proportions? Discuss which approaches and sample sizes worked best for different information needs (e.g. uncovering different types problems vs. estimating proportions of problems). Talk about how moving from candies to sampling with real people may impact the results- which can move you into a discussion about non-response bias, how people with different issues may respond differently to you data collection, and cost issues.

Note: If folks can see what they’re pulling, they may bias the random samples- try having one person hold the bag, and another pull with their eyes shut. Also, don’t start eating until you’ve pulled the last sample- otherwise your pre-set proportions will get thrown off!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· ·

7 comments

  • Buyana Ulzii · June 29, 2010 at 9:28 am

    Thanks Nicole! I really like this candy approach. I am a new member here and just read your tip. It’s never too late for appreciation.

    Reply

  • Admin comment by Susan Kistler · March 3, 2010 at 8:37 am

    Michael, yes, have used beans myself – and they are certainly better for the waistline. I used them in particular when working with a group that focused on health issues.

    Reply

  • Nicole Vicinanza · February 25, 2010 at 12:37 pm

    Susan- thanks for the qualitative excercise tip- I’ve used this one before and found that it works particularly well with community organziations that have gotten overwhelmed with qualitative data from client surveys. This really helps them see different patterns they can share with stakeholders, and break out of the rut of either listing all the responses or ignoring the qualiatative part of their data all together!

    Reply

  • Nicole Vicinanza · February 25, 2010 at 12:29 pm

    Hi Michael:
    Thanks for the great suggestion!–Nicole

    Reply

  • Paula Bilinsky · February 25, 2010 at 9:11 am

    Ever since Nicole taught me this technique, I have used it many different times all over the world with all different types of training participants. It is almost always the most highly rated part of my training. It seems that candy is the universal training aid!

    Reply

  • Michael Crow · February 24, 2010 at 11:44 pm

    Thanks for the great explanation and tips. Along with a collaborator, I deliver a training course in basic statistics for performance measurement/evaluation, with a target audience of environmental regulators. We have developed a similar exercise — although with dried beans, which will hopefully be a little bit better than candy at maintaining population size over repeated samples. 🙂

    If folks are looking to simplify a little bit, we focus in our course on survey results for proportions based on simple yes/no questions, and one color bean represents “yes” answers and another represents “no.”

    Reply

  • Admin comment by Susan Kistler · February 24, 2010 at 5:32 am

    Thanks Nicole! I hadn’t tried this approach. Another way of using candies to convey some basic analysis techniques is when working to explain qualitative analysis. We would provide teams with a bag of assorted candy (kisses, small wrapped chocolates, jelly beans, hard candies, tootsie rolls, etc.). Each bag would have the same assortment. We’d then ask the teams to group the candies using any schema that they felt was appropriate and to do a quick report out to the group. Across groups, we’d end up with those who did permutations of chocolate/non chocolate, wrapped/unwrapped, hard/soft, etc. Sometimes we’d also have them group and then tell them to start over and group using different categories or we’d ask them to group into two categories then four then more. Ultimately, the goal was to convey that there wasn’t one perfect way to group concepts for qualitative analysis, but that you were trying to find one that readily told the story of what was there, assisted with reporting, and made sense in context.

    Reply

Leave a Reply

<<

>>

Archives

To top