Hello readers! I’m Sheila B Robinson, aea365’s Lead Curator but I’m also an evaluator and consultant. I often write about what’s going on at AEA, but today I’m returning to my roots and sharing a bit about data collection. Almost a year ago, I transitioned from full time employment with an organization and part-time freelance work to being a full-time independent consultant (my company is Custom Professional Learning, LLC). I’ve been spending much of my time on the road teaching workshops, but my evaluation projects seem to have me in a constant state of data collection and I enjoy reflecting on and writing about this work.
Lately, I’ve been conducting a lot of focus groups and honing my skills as a protocol planner and focus group moderator.
Lessons Learned:
- Know your respondents. My recent focus groups have been with public school teachers. Having worked in schools my whole career I know that participating in focus groups moderated by strangers from a local university before or after school hours would not hold a lot of appeal. To get willing participants my evaluation team enlisted the help of a committee we had been working with on the evaluation. We asked them to serve as ambassadors for the project, talk with their colleagues about it, issue the invitations for focus group, and make the arrangements at times and places convenient for participants. We have held three successful focus groups this way thus far, with more on the horizon. We would never have been this successful “cold calling” (or emailing).
- Training moderators is important. We had a situation where teachers were very willing to participate at a particular place and time, but there were more than 20 of them in the room – way too big for one focus group. In this case I was the only evaluator and the only one with focus group experience, so I enlisted the help of a couple of volunteer moderators and we split the group of teachers into three smaller focus groups that ran simultaneously. Moderators need to be especially comfortable using probes – knowing when and how to use them effectively to draw out participants and dig deeper to understand their feelings, opinions, attitudes, experiences, or whatever we are measuring. Even more important, they need to know how to manage the voices in the group, navigating skillfully to draw out quieter voices and limit potential conversation dominators.
- Use survey data to inform the protocol. In many cases the evaluation project has surveyed individuals prior to the focus groups, and the resulting data can make excellent fodder for focus group prompts and questions. I examine the dataset and see where I still have questions – perhaps it’s “Why did people respond this way?” or “What more do I need to know about this to answer the evaluation question?”
Rad Resources: I’ve been writing about data collection with focus groups and surveys for Frontline Education, Frontline Research and Learning Institute, and BlueSky eLearn.
There are a lot of great focus group texts out there, but one of my favorites is this one, by Robert Kahle: Dominators, Cynics, and Wallflowers: Practical Strategies for Moderating Meaningful Focus Groups
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.