AEA365 | A Tip-a-Day by and for Evaluators

TAG | online focus groups

Hi! We are Laurene Christensen and Vitaliy Shyyan from the National Center on Educational Outcomes. NCEO is a federally-funded technical assistance center that works primarily with state departments of education to help improve outcomes for students with disabilities, English learners, and English learners with disabilities.

Our work often involves a national reach, including gathering information from a variety of stakeholders across the nation. That’s why we like to use online, asynchronous focus groups to gather information from stakeholders.

Hot Tip: Spread focus groups out over 3-4 days. By having a short time frame for the focus group, participants will keep the momentum going. It is important to recognize that participants will interact with the online focus group questions at the time of day that is easiest for them, so some will respond in the mornings and some will be more active with the questions in the evenings.

Hot Tip:Make sure your moderator is highly engaged. Encouraging comments and asking follow-up questions to the participants’ posts keeps everyone coming back! Participants want to know that their contributions are appreciated and an engaged moderator will know how to acknowledge them.

Hot Tip: Post only two or three questions per day. This number of questions will keep the posts manageable for participants. But, posting daily is important to keep them coming back and engaging with the focus group site. Bringing participants back to the site regularly adds to the quality of the focus group results.

Hot Tip: Encourage participation by having participants maintain anonymity and participate voluntarily. In our focus groups, we often use tree names for pseudonyms because most tree names are gender neutral. For example, in one focus group with participants from Arizona, pseudonyms included “Saguaro” and “Palo Verde.”

Lesson Learned: Online focus groups have an advantage in that at the end, you will have a transcript of the focus group without the need for a transcriber. For data processing, we use qualitative analysis software to code the content and identify emerging themes. Although preliminary findings come primarily from immediate responses to the topical questions, we found that instrumental information can also be generated through additional comments and follow-up questions from participants.

Have you used online, asynchronous focus groups in your work? Tell us more in the comments!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! We are Manolya Tanyu, senior researcher, and Nicholas Read, researcher at American Institutes for Research (AIR), a behavioral and social science research organization, sharing our insights on conducting virtual focus groups as part of a national evaluation of enhanced youth mentoring programs. We’re using videoconferencing as an alternative to in-person focus groups to save time and travel costs.

Lessons Learned: Virtual focus groups feel different than in-person focus groups, where the researcher is in the same room with participants and can observe group dynamics. In some of our focus groups, participants from the partner agencies met in one location and shared a computer with a webcam. This format allowed them to interact with each other but in most cases made it difficult for our remote facilitator to see everyone in the room. In other focus groups, participants joined separately via different computers. While this format allowed the facilitator to see all participants, it limited group interaction and participants tended to respond to the facilitator rather than participate in group discussion. In some instances, participants lost internet connection and were only present via teleconference, eliminating visual observation altogether. Overall, virtual focus groups can be an effective way to collect qualitative data, with a few parameters:

  • all participants in the same room (if possible)
  • reliable internet connectivity
  • test the technology
  • clear guidance to participants that focus groups should be group discussions.

Hot Tip: We used Cisco’s WebEx online conferencing (one of many options). WebEx allows participants to join via computer without having to download software, has both telephone and computer-based audio options, and allows video and audio recording. We obtained consent from each participant to be recorded (they signed and email us consent forms), recorded each session for transcription, and also had notetakers in case of technical glitches. Coordinating each virtual focus group took about five hours, including finding a date / time that worked for everyone, setting up WebEx, sending e-mail invitations, scheduling and conducting a test run, retrieving video and audio files from WebEx, sending audio for transcription, and cleaning up our notes.

Cool Trick: We highly recommend conducting test runs with participants before the actual focus group. For the majority of participants, this was their first time participating in an online focus group using videoconferencing. Test runs helped participants sort out technical issues and test audio and webcams. A 15-30-minute test run can save valuable time during the actual focus group.

Rad Resources:

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Beth Robelia. I am the proprietor of a micro-business that focuses on STEM education research, curriculum development and program evaluation.

In evaluating the effectiveness of social media, the disbursed nature of users may make face-to-face focus groups impractical. Online focus groups can bridge the gap in geography by placing participants in an online forum. Online focus groups may draw out less assertive members by giving everyone a chance to respond to a question and reduce intimidation by perceived differences in social capital.

We used online focus groups to evaluate the impact of the Facebook application on young adults who participated in a climate change newsgroup.  Each participant chose a screen name so that identities would not be revealed. We posted guidelines for participation to encourage respectful communication. The asynchronous group lasted three days. Participants were asked to log on at least twice a day; once to answer main questions and again to answer follow up questions posted by evaluators asking for more information about issues raised in earlier posts. We created transcripts by downloading the text from Yuku. The discussion and comments were helpful in evaluating the Facebook application and enriched our qualitative research.

Hot Tip- Create a password-protected site: This helps participants feel safe. Limiting access helps ensure data privacy. We set up a password-protected group in Yuku.com.

Hot Tip- Consider the need for anonymity: If participants have been anonymous in the social media site you are evaluating you should consider continuing anonymity. You may still want to consider anonymity for honesty in answering the questions and compliance with research protocols.

Hot Tip- Write questions with the methodology in mind: Asynchronous communication via text is different than the informal exchange that may occur during a verbal discussion. You may want to have participants log in twice a day to respond to other’s comments.

Hot Tip- Warning!: Online focus groups create a tangible online artifact of the discussion that should be considered in ethical deliberations.

Rad Resources:

Burton, L., & Goldsmith, D. (2002, June). The medium is the message: Using online focus groups to study online learning. Paper presented at the  Association for Institutional Research, Toronto, Ontario.

Steward, K., & Williams, M. (2005). Researching online populations: The use of online focus groups for social research. Qualitative Research, 5, 395-416.

Watson, M., Peacock, S. & Jones, D. (2006) The analysis of interaction in online focus groups. International Journal of Therapy and Rehabilitation, 13(12) 551-557.

The American Evaluation Association is celebrating Minnesota Evaluation Association (MN EA) Affiliate Week with our colleagues in the MNEA AEA Affiliate. The contributions all this week to aea365 come from our MNEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Archives

To top