I’m Sue Hamann from the Needs Assessment (NA) TIG. I have worked as an Evaluator for more than 40 years, currently employed at the National Institutes of Health as a Health Scientist and Science Evaluation Officer.
I’m writing about how review boards can be helpful in valuing human (program) participants (beneficiaries), thus promoting ethical standards, and our role in this as professionals working in needs assessment (NA) and evaluations. One of our foundational texts, Planning and Conducting Needs Assessments (1995, Witkin and Altschuld), reminds us that service recipients “are those for whom the system ultimately exists; they are at the heart of the NA process” (p. 11). I often see reluctance by evaluators, program funders, and others for this inclusion. Sometimes funders and others are interested, but the evaluator is not accustomed or not willing to work with an Institutional Review Board (IRB), described here by the American Psychological Association, or other review board. Guidelines, laws, and regulations address this engagement. Review boards can be our allies in helping us conduct the most ethical assessment or evaluation possible.
I’ll give you an example from my own work in NA and evaluation. My agency undertook a series of initiatives to change our culture and climate. The initiatives included trainings and interactions with staff, as well as data collection from staff, that was unusual for us. I considered the employees to be human (program) participants. Moreover, I considered the series of initiatives to be research, and therefore it was appropriate to involve an IRB, knowing that the agency leadership intended to generalize the results beyond our walls and publish results. Some program staff and evaluators disagreed with me. Having been a member of the IRBs at two institutions, I know that people often want to avoid their IRB, and I know that the IRB makes the determination of whether a project is research and, if so, whether it is subject to or exempt from IRB jurisdiction.
I submitted our project to our IRB. It was judged to be research, and an annual report was requested. Importantly, the IRB team had some great ideas about how to protect the employees in ways I had not considered. My work was made stronger because I engaged with the IRB.
Have you ever heard the phrase, “It is better to seek forgiveness than to seek permission”? A panelist in an evaluation webinar said it recently, with a smile and a shrug. The idea, of course, is to sidestep certain procedures and pretend ignorance. Why would professional evaluators do this? We are leaders in the field, great advocates for social justice, and we should be role models about adherence to rules, regulations, and principles that have been accepted and promoted by AEA or that are mandated for our use.
The inclusion of program participants and beneficiaries improves the utility of NAs and evaluations. Actively working with IRBs and other review boards can also improve propriety. Review boards need not be the domain of large agencies or academia. If you work in a smaller organization and would value what independent reviewers could offer, you might consider setting up a small review body to test out what works, and then build from there. This Peer Review Process Handbook from the U.S. Bureau of Safety and Environmental Enforcement provides a number of useful tools if your organization is getting started in thinking about setting up a review panel.
Rad Resources
AEA Guiding Principles (See Principle D: Respect for People)
PROGRAM EVALUATION STANDARDS OMB M-20-12, Phase 4 Implementation of the Foundations for Evidence Based Policymaking Act of 2018: Program Evaluation Standards and Practices. (See Section on Ethics)
Yarbrough DB, Shulha LM, Hopson RK, Caruthers FA. The Program Evaluation Standards 3rd Edition. Sage Publications, Los Angeles. 2011. (See chapter on Propriety Standards)
The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the NA AEA Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.