I’m Jim Altschuld, an AEA charter member who’s written about needs assessments (NAs) for nearly 4 decades and Professor Emeritus, The Ohio State University. So why aren’t journal articles about NAs plentiful? Possible answers:
- Journal reviewers don’t like NAs, are biased, and reject most articles.
- NAs are for internal organization and agency purposes, not for publication, the question isn’t pertinent.
- As NA practitioners, we don’t focus on publications.
*D. Manuscripts are rejected due to weaknesses in instruments and methodological problems.
Why choose *D? Most NA books are primarily ‘how tos’ and don’t delve beneath the surface into needs assessment methods and the validity of the data collected.
Can we draw legitimate conclusions and implications if validity isn’t there? If we do, would the results be tainted – The Fruit of a Poisonous Tree? I raised 14 issues about double (what should be, what is) scaled NA surveys in a 2018 Ignite session, and discuss two issues (survey pre-structuring, failure to categorize) below.
Issue 1 -Skewness coming from ‘What Should be’ Items on NA Surveys (Pre-structuring):
Needs assessors most often do qualitative studies, review the literature, and consult with others to learn what’s important before developing ‘what should be’ items for stakeholder surveys. What’s wrong with that?
Well, items are frequently rated on the high end of the scale leading to negatively skewed results with item values that are very close and not differentiated from one another. Little information is gained. What do we learn from this kind of data, how meaningful are the what should be scores, and what sense do we make of discrepancies/gaps against them as anchor points? I’m not sure.
Issue 2 – Failure to Categorize:
Say we have 40 double scaled items about a topic that can be put into 3-4 categories. Besides rating the items, if respondents ranked the categories, undoubtedly they wouldn’t be of equal rank. Logically, high ‘what should be’ items in the top ranked category would be more valuable than similar high items in a lower category. This offers additional insight and information about the skewness. Analysis/interpretation of data becomes complicated, but the understanding gained would be worth the cost. Besides that, there’s likelihood that the validity of the instrument (at least face validity) would be improved.
Hot Tips:
- Issue 2’s discussion also applies to issue 1 and important for better instrument design.
- Dig into the literature (see 3 references below) for subtle dimensions of methods used in collecting NA data. This will improve the validity of assessments and provide findings that are more sound and acceptable for decision making. Only then will we achieve the standards of professional journals.
Rad Resources:
- Altschuld & Witkin (2000). From needs assessment to action: Transforming needs into solution strategies. (See, Chapter 3, Data-related issues and survey methods.)
- Hamann, M.S. (1997). The effects of instrument design and respondent characteristics on perceived needs. Unpublished dissertation, Ohio State. (Appendices A and B on category rankings.)
- White & Altschuld (2012). Understanding the “what should be condition in needs assessment data.” Evaluation and Program Planning.
The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Thanks Jim for a great blog on publication issues with NAs. I agree that it’s become harder and harder to publish any evaluation data since many reviewers don’t see this as generalizable knowledge and so the questions of ‘so what?’ and ‘who cares? are harder to answer outside of the group for which the work was done. I teach a # of workshops on scientific writing; I’ve published a lot; and I review for 15+ journals – and the bottom line is typically what does this add to the literature? Without answering that question (plus the ‘so what’ and ‘who cares’ or ‘who should care’), reviewers are hard-pressed to vote in favor of a publication. Even a rigorously done RCT may not add anything to the literature despite the fact that the data are highly valid and reliable (something that evaluation data is hard to demonstrate). So what can we do? While we strive to demonstrate that the data has both validity and reliability (and, yes, we have to admit the potential biases of any study – even an RCT), we have to make clear what this new work adds to the literature and why it needs to be ‘out there’. In reviewing manuscripts, I’m often left with just using my own sense of what the utility is of this study – without the authors being able to describe that themselves. Yes, there are some evaluation-specific journals that we can submit these manuscripts to whereby reviewers are more keen on the pro’s and con’s of evaluation work, we often want to get to a different and more broad audience with the story we’re telling. It would be good for an AEA conference to bring together a panel of journal editors and/or journal reviewers and pick their brains about how best to get our messages out there and showcase the value of what we’ve learned and can teach others through spread. Again, glad that you tackled this for today’s blog.