Hello again! I am Beverly Peters, assistant professor of Measurement and Evaluation at American University. This is the fourth article in a 6-part series on Using Focus Groups for Monitoring and Evaluation. The third article discussed populating a focus group—a very important yet challenging aspect of focus groups. This article will discuss an equally challenging and important part of focus groups—developing focus group questions and a questioning route.
Let’s go back to the purpose of a focus group: To facilitate the cascading of conversation amongst participants. As you plan your focus group, you should consider both the kinds of questions that will facilitate conversation, in addition to the order to ask them—or what researchers call the “questioning route.” Always pilot your questions and questioning route with colleagues or others who are knowledgeable about the project and its local or organizational culture.
A first step is to devise questions that are easy for participants to understand. Make sure that your questions are not leading or judgmental, confusing or double barreled. Your questions should also reflect emic interpretations of language and culture.
A good interview question is not always a good focus group question, however. Kruger and Casey tell us that we should write focus group questions so they are conversational, open ended, and facilitate thought amongst participants. Try to avoid questions that will result in individual answers—remember that the purpose of your focus group is not to collect information from individual participants, but rather, to facilitate conversation amongst them. Ask questions that are conversational in nature and get people talking to each other—perhaps to flesh out a puzzling issue around the project—instead of just talking to the moderator.
I like starting with a grand tour question such as “Tell me about….(insert something around the project.)” With this type of open ended question, I can gauge the tone of the room upfront, identify strengths or issues related to the project, and very importantly, tailor subsequent questions to reflect these as needed.
I also like to ask questions that require the group to discuss and rank issues—for example, asking the group to identify and then rank the challenges the organization faces implementing a project. I also like using ideal scenario questions and participatory tools such as decision trees to help facilitate conversation. For a 90 minute focus group, I usually craft 8-12 conversational questions—if your questions are truly conversational you should not need more than this.
After you have well thought out questions, you should consider their order. Your questioning route should be logical and flow like a natural conversation. You should sequence your questions so they focus more and more specifically on the topic of study over time. I usually start out general and open ended, and over time focus in on the topic and even the controversial aspects of it. This helps participants feel comfortable with the discussion, and open up to share opinions and controversies around it.
Look for Parts 5-6 in the coming months!
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Beverly,
Thank you for the entire series on Focus Groups. I have found it very helpful for both my course work as a Graduate Student at Queen’s University as well as within my role as Learning Services Director in my school division. There seems to be a shift in my school division to include more focus groups and stakeholder engagements yet most of us in my department have much, if any, training in organizing and running focus groups.
While I had thought that creating the focus group questions in advance would be important I hadn’t really considered creating a question route for the focus group. It makes sense to start with some easy questions that are designed to generate conversation and put people at ease before digging into some of the more difficult questions a program evaluation may tackle. While I agree a question route would be very important, I wonder under what conditions it would make sense to deviate from the route? Is it ever ok? Or should a focus group leader “stick to the script”? I think there could be some unexpected things that arise during a focus group discussion that might warrant a deeper dive, however, it would be easy for an unskilled or inexperienced focus group moderator to have the conversation spiral out of control and get way off track. I wonder how a focus group can be designed to allow flexibility to get to unintended, but important places, without getting off track and rendering the focus group ineffective.
My team has decided to tackle this issue my using a Focus Group Lead Team. The roles are: Moderator, Recorder, and Pacer. The role of the Pacer will be to keep the focus group questions moving along based upon time, but also based upon the question route. They will allow for some divergence from the route, but if they see things getting too far away they will cue the moderator and hopefully things will get back on track. I am not sure if this is the right approach, but it seems to allow us the opportunity to have someone move the focus group forward if the moderator loses track of where things are going or ends up following questions down a rabbit hole.
Again, thank you for the series on Focus Groups. I found the posts helpful to me as I prepare to support the development of focus groups in my school district as well as with my graduate work focused on Program Evaluation.
~Kevin
I am new to evaluations and I read your article with a lot of interest because I am realizing how much of an art there is to evaluation. I am interested in focus groups as a way of collecting data as it seems to be quite interactive and offers an opportunity to deeper insight than perhaps an electronic survey might.
It makes sense to me that questions would be designed prior to engaging in a focus group. However, I had no idea that ordering the questions in a ‘questioning route’ to facilitate and narrow the focus from general to specific would be a part of the process. It reminds me of a strategy that counsellors use when getting to know clientele. Starting from the general, conversational pieces to becoming more specific as trust as built in the process
I am wondering how intentional the choice of pilot people would be in testing out the questions that would be used? Do you ask for a few volunteers to assist in determining the validity of questions that will be asked of the focus group? As well, would there ever be a moment that these ‘pilot’ people’s responses might be used as information in the evaluation if their responses led to a deeper understanding of the program or item that was being evaluated?
It sounds as if the intent of a focus group, as you describe, is to create conversations and the questions may just be the catalyst for those moments where information can be naturally gathered. In which case, focus group moderators need to have a significant arsenal of potential questions that can be flexibly used.
If the initial question suffices in creating conversations that the moderator finds relevant to the evaluation, could further questions serve to frustrate participants? Does the role of the moderator become one of recorder, instead of questioner if focus group participants seem to be leading themselves?
Thank you for writing such a thoughtful series of articles. I am hopeful that I can find the answers to my many questions and look forward to your response.
I am relatively new to the field of evaluation and the various methods used. I am currently enrolled in a course focusing on program inquiry and evaluation. We were asked to visit AEA365 and comment on an article we found interesting. I happened to read over yours and found your article on focus groups quite insightful. You bring up some very great points; I believe it is extremely vital to ensure questions asked during focus groups are open ended, as a means to foster discussion and conversation amongst participants. Through these conversations, one is able to gain useful information on the group’s feelings and opinions. The only concern, or rather, question that comes to mind is whether participants may feel pressured to agree with a dominant view point and conform to specific opinions or beliefs the rest of the group may have. Are there ways to avoid this? How can this be mitigated?
Hello Beverly. I am a Masters of Education student at Queen’s University in Canada. I am currently taking a course in program evaluation. I have found your series of posts about focus groups to be very helpful. You have provided an easy to follow outline for new evaluators. I agree with you that focus groups are a powerful way to collect qualitative data by generating insider perspectives through conversations. Evaluators are able to get rich descriptions of experiences through a focus group that they wouldn’t be able to get through other methods. I appreciate your advice about how to set up the physical environment for the focus group as it is such an important factor in contributing to its success. Your advice to make multiple smaller groups if there is more controversy, seems valuable and something I hadn’t considered.
Your suggestion to consider separating focus groups by age, sex, language, income etc. surprised me. I do understand the reasons for this is to make people comfortable in sharing their experiences and you provided a great example. I am wondering in your experience, if the act of separating people into these groups has ever generated conflict or negative feelings towards the evaluator?
Your advice to create a list of questions and then organize them into an order ‘questioning route’ that would be best to facilitate conversation is very valuable. I think having the focus group rank issues is a good way to generate discussion and identify easily what the major issues are with the program. I am wondering how you address the group if ranking creates polarization.
Thank you for your providing such relevant information. As a teacher studying evaluation techniques, I find your posts extremely helpful.