We’re James W. Altschuld, Hsin-Ling (Sonya) Hung, and Yi-Fang Lee from Ohio State, Virginia Commonwealth, and National Taiwan Normal Universities and have worked many years on needs assessments (NAs). We’re providing tips to enhance validity for data from qualitative techniques – Focus Group Interviews (FGIs), The Nominal Group Technique (NGT), and double scaled survey responses.
FGIs (Use of Facilitators with Special Rapport)
Prominent author Richard Krueger’s work discussed training community members to lead FGIs. From his research, facilitator rapport was the key to success in using standard FGI procedures. In 1991, Yoon employed FGIs to study needs of international Asian students at a major university. In a females-only group, a sensitive issue emerged and analogous to Krueger’s work, with a female leader with strong rapport to the participants.
Hot Tip: FGI facilitator and rapport are important for obtaining valid, meaningful NA data and enhance information about needs.
The Nominal Group Technique (A Group in Name Only)
Nominal Group Technique (NGT) is frequently implemented incorrectly. The NGT looks like a regular group discussion, but it is a clever brainstorming procedure with precise rules and steps. For most of the NGT’s steps, no interaction between participants is allowed (in normal groups, people interact). A facilitator enforces the rules subtly and lets interaction only occur close to the end of the process, not before that. If participants interact in the beginning phases, it’s a discussion group, not an NGT and NGT’s restrictions have been violated. ‘Nominal’ in the title has a clear meaning and implication.
Hot Tip: Review the literature on NGT and adhere to the rules (Witkin & Altschuld, 1995), thus yielding better data.
Qualitative Follow-up of Surveys (Explore what you don’t know from survey data.)
Many NA surveys with single or double scales are administered to stakeholder subgroups to determine if they agree or disagree. When they disagree, why do they differ? Without more knowledge, the needs assessor speculates about the underlying thoughts of the subgroups and cannot really interpret the data. Lee and colleagues (2007) addressed this concern. For group differences on a double scaled survey, they sent a second, brief visual type of survey to the groups asking why they responded as they did and why the other group disagreed. The figure below contains a description of the highlighting of the group difference on the S scale.
The need was the gap between S (satisfaction with) and I (implementation of) a service, both measured on 5-point scales. The bars and the ellipse were marked in different colors. The follow-up was helpful in interpreting and adding depth and validity to the results.
Hot Tip: This approach requires more work for the assessor but how else could you make meaning out of the data?
Rad Resources:
- Effects of multiple stakeholders in identifying perceived needs. Evaluation and Program Planning. Lee, Altschuld & White, (2007).
- Planning and conducting needs assessments: A practical guide. Witkin and Altschuld (1995).
- Yoon’s 1991 study described in Altschuld and Witkin, (2000). From needs assessment to action: Transforming needs into solution strategies.
The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.