I am Melissa K. Demetrikopoulos, Chair of STEM Education and Training TIG and Director of Scientific Communications at the Institute for Biomedical Philosophy here with Terry Mills of Morehouse College and BlueSKY Collaborative Partners. We are presenting information about an Exploratory Sequential Mixed Methods approach being utilized with our partners at Morehouse College including J.K. Haynes, Cynthia Trawick, Jigsa Tola, and Triscia Hendrickson.
The Study
Phase 1 involved data collection through an extensive survey on participant knowledge of, and success with, a variety of federal funding opportunities serve Historically Black Colleges and Universities (HBCUs) as well as National Science Foundation (NSF) Research on Learning in Formal and Informal Settings (DRL) funding mechanisms not targeted to any specific population.
Phase 2 utilized semi-structured focus groups where participants were encouraged to share best practices implemented, as well as ongoing challenges, with obtaining funding. Focus group questions built upon data obtained through surveys and allowed us to explore issues revealed. Guided discussions explored engagement of HBCUs faculty and administrators with STEM education research across DRL funding and more broadly across NSF funding. Focus groups were conducted with homogenous and heterogenous groupings based on participant titles within academia. Focus groups did not aim at consensus building, but were structured to hear all voices, explore differing perspectives, and provide opportunities to change their minds or revisit prior topics.
Recurring topics were identified presenting best practices and a number of challenges which hindered implementation of these best practices. Terry Mills is a sociologist and suggested a methodology that is commonly used in sociology but less commonly utilized by STEM researchers: the use of scenarios or vignette-based interviews that would bring out the potential richness of qualitative data and provide a mechanism for participants to discuss their perceptions and experiences related to these topics.
Phase 3 made use of generalized fictionalized scenarios developed from information shared during focus groups. Seven scenarios were originally drafted using ChatGPT AI with input from focus group responses. Once drafted, five scenarios were chosen and edited to reflect questions under investigation.
Participants were presented with titles of the five scenarios and asked to choose three scenarios they were most interested in speaking about. Each scenario was explored with a series of guiding questions in a semi-structured interview. Rather than having participants comment on what individuals did or might do next, we utilized a common approach to scenario use and asked participants if their institution had experienced something similar. The scenario was explored with a series of questions designed to understand who was, or should have been, involved; what the benefits were or might be; best practices; and potential challenges or hurdles. Following each scenario, a number of possible challenges that were identified during focus groups were presented and participants were invited to share if and, if so, how these challenges may apply at their institution as well as to make suggestions for best practices on how they have, or could be, overcome. This approach allowed participants to both discuss practices at their home institutions as well as to convey their ideas and perceptions about idealized best practices including suggestions for ways to overcome potential hurdles.
Rad Resources
Sampson, H., & Johannessen, I. A. (2020) and Murphy, J., Hughes, J., Read, S., & Ashby, S.(2023) do a good job of giving context to use of vignettes in qualitative research.
This project was funded through National Science Foundation DRL-2131762: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.
The American Evaluation Association is hosting STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to AEA365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.