The Evaluation Center at WMU Week: In Defense of Checklists: More Than Checking a Box by Lori Wingate

Hello! I’m Lori Wingate, director of The Evaluation Center at Western Michigan University. In 1999, less than two years after starting my job at The Evaluation Center, I was charged with developing and curating an online collection of evaluation checklists as part of a National Science Foundation-funded project to support STEM evaluators in their practice. The Evaluation Checklist Project has endured, and we now have 30+ checklists on a wide variety of topics. I’m writing this blog in defense of checklists as a to promote excellence in program evaluation.

Checklists sometimes get a bad rap. I’ve heard smart people with so much wisdom to share claim they can’t reduce their advice to a simplistic checklist. But who said checklists have to be simplistic? Checklists aren’t about dumbing down complex content or simply “checking boxes.” They are about keeping important issues on the radar, avoiding common mistakes, and calling attention to matters that may be otherwise overlooked. At their best, evaluation checklists distill the best of evaluation theory, well-established practices, and—in many cases—many hard-won lessons learned into succinct statements about things evaluators should do, or at least consider doing, to deliver high-quality services. Here are some examples:

• Have you ever seen an evaluation plan loaded with positively biased indicators so the evaluation is set up to only look for positive program effects? If the evaluation’s designers had used the Checklist of Criteria for Selecting High-Performing Indicators, they would have known to use nondirectional language when defining what will be measured in an evaluation.
• When reviewing a transcript of an interview you conducted, have you ever been surprised (even embarrassed?) by how long-winded your questions were? Have you ever been in a focus that just went badly? These unfortunate situations can by avoided by using the Checklist of Effective Communication Strategies for Interviews and Focus Groups.
• Have you ever come across an evaluation report or other resource online you wanted to cite, but the document didn’t identify an author or publication date, making it challenging to figure out to include in your reference list? This wouldn’t happened if the authors used the Checklist for Program Evaluation Report Content.
• Have you ever worked with a group that had trouble developing evaluation questions that really hit the mark? The Evaluation Questions Checklists offer essential reminders about the purpose of evaluation questions and the attributes of effective evaluation questions.

Checklists like these aren’t just for novices. In fact, the more comfortable professionals (in any setting) get in their practice, the more likely they are to be overconfident that they’ll remember all the nuances of good practice and be able to recall that information when needed most.  

Rad Resources

• The Checklist Manifesto by Atul Gawande provides compelling accounts of the power of checklists to safeguard against common oversights and mistakes in all types of professional practice, with a focus on the Safe Surgery Checklist.
• No surprise, the raddest resource on evaluation checklists is the Evaluation Checklist Project website. Admittedly, some of the checklists need updating, and the coverage of topics isn’t comprehensive. (Keep in mind, Evaluation Center staff basically do this work in their spare time). But they are also full of excellent insights from a wide range of experts.

Before closing, I want to note The Evaluation Center’s appreciation a grant we recived from the Faster Forward Fund in 2017 to revamp the Evaluation Checklist Project website, develop a project charter, create a checklist style template, convene an advisory group, and codify checklist development and validation procedures.

We welcome proposals for new checklists – contact at me at lori.wingate@wmich.edu to start a conversation.

---

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.