We’re Sonya Hung, Shea Wenzler, and Jim Altschuld from Virginia Commonwealth University and Ohio State. We’re sharing our experience on conducting online nominal group technique (NGT).
The Setting
We had a small project to do limited evaluation capacity building for faculty (mentors) and mentees in Underrepresented Minority (URM) Students’ training programs for a variety of STEM (science, technology, engineering and mathematics) and health science fields. After these remote trainings were completed, we conducted NGTs via Zoom, with small groups of involved faculty and student participants to obtain ideas and input for designing a uniform instrument to be used across all such mentor/mentee relationships. Participants joined the meeting from their offices, homes, or other locations best for them.
The NGT Procedure
The NGT is not exactly a group procedure for over half of its conduct. When done in person, it follows specific rules and the group does not interact until a certain point in the process. In other words, it looks or appears to be a group, but it isn’t one. It is subtle. Therein lies the question—could we accomplish this subtlety over the web? Here’s what we did:
- Developed thought-inducing questions and sent them in advance of the NGT sessions. We asked participants to reflect on “the most important factors that contribute to a successful mentor-mentee experience” and briefly describe at least five or more. For some, we also had a negative version of this probe.
- Scheduled the electronic NGTs. Most were quite limited in number, 2-4 persons, since personal schedules did not easily mesh together.
- Conducted all the NGTs in accord with standard procedures. We followed the following steps: Explanation of the rules, silent brainstorming, group round robin (with no discussion or rationale being solicited for their ideas), consolidation of the list, and individual selection of their top choices from the list. For details on how to implement the steps see Witkin and Altschuld’s book. Note, no discussion was permitted or allowed during the session
- Reviewed with participants their perceptions of what was done. At this point, we stressed that by not letting discussion occur, the basic not a group dimension of the NGT was preserved. Otherwise it would have been a true discussion group, not a nominal (in name only) one.
The Yes’s and No’s: Our Thoughts
Yes: We administered the NGT over the web and obtained valuable input for the instrument. Participants in this project did follow the rules pretty well. Though there were one or two instances where they wanted to explain or discuss their ideas during the generation and round robin phases, we were able to contain or curtail such occurrences. So, it did work.
No: Having done this in a lot of classes where a small group simulates the NGT and the rest of the class observes, we asked two questions during an overall debriefing. First, “What did you observe?” This question was met with answers like, a neat brainstorming process, lots of ideas in a short period of time, everyone is involved, and so on. The second question, “What did you not observe?” was usually met with silence and confusion. Sometimes, somebody would say, “I noticed that there was no interaction between group members.” BINGO. It looks like a group, but it isn’t one!
Lesson Learned:
We obviously think the technique works with web-based applications and recommend its use, but recognize that the visceral classroom impact probably does not generalize as well.
The American Evaluation Association is hosting Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to AEA365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.