Hi, we are James W. Altschuld, Hsin-Ling (Sonya) Hung, and Yi-Fang Lee from The Ohio State University, Virginia Commonwealth University, and National Taiwan Normal University, and have presented on, written about, and been involved in NAs for many years. In reviewing manuscripts for publication, we have observed measurement and analysis issues in double scaled surveys used for identifying discrepancies between what should be (importance) and what is (current) conditions. They include wording, scaling choices, organization, misleading conclusions and interpretations, and others.
If the survey is not sound what comes from it will be the fruit of the poisonous tree. By offering ideas for improving NA surveys some (not all) of the weaknesses can be attended to and the quality of needs work will be enhanced.
Hot Tips:
- Read sources about the design and especially strengths and weaknesses in NA surveys (some sources do both, see Rad Resources).
- Search for sample NA surveys in the area of concern, examine/critique for strengths and weaknesses as noted in point 1.
- See if other techniques have been used with the surveys. (Having multiple sources of information is good practice.)
- Conduct pre-interviews (focus group, individual) with a few respondents regarding their thoughts and the language they use for the area of concern. (Will make the instrument more meaningful.)
- Cluster items into sections and consider having respondents rank them after completing the survey. (Not all clusters will be equal of value.)
- Employ options like don’t know (DN), no information (NI) upon which to decide, not applicable (NA), etc.
- Forcing choices without such options, as in point 6, may produce misleading data and additionally the options provide useful information.
- Have an undecided (neutral) response on the scale. (Similar rationale to point 6.)
- Consider alternatives such as magnitude estimation scaling (MES), fuzzy scales, rank ordering approaches, etc. (Let’s be expansive and innovative in what we do.)
- Multiple ways exist for analyzing data ranging from simple/weighted needs indexes, means difference analysis, proportional reduction in error (PRE), etc. (Try several, see if results differ and conclusions are affected.)
Lessons Learned:
- Seemingly simple double scaled surveys are not so seemingly simple.
- ‘There are 95 rules of survey design and after the 95th there are 95 more you don’t know about.’ This applies doubly to double scaled NA surveys and to illustrate the assertion, note that we haven’t touched the gnarly topic of “how to word” items.
Rad Resources:
The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.