CP TIG Week: What Moments Were CRITICAL? Reflection in Evaluation by Jonathan Scaccia and Paul Howard

Hi, we’re Jonathan Scaccia of the Wandersman Center and Paul Howard of the Institute for Healthcare Improvement (IHI), and we’d like to share a method that we’ve been using in our evaluation work for the past 4 years: The Critical Moments Methodology.

Lessons Learned: Community psychology puts a strong emphasis on participation and inclusion in all stages of a project; from design, to implementation, to evaluation. It’s natural that those who identify as community psychologist gravitate to participatory and Empowerment Evaluation methods.

Spreading Community Accelerators for Learning and Evaluation (SCALE), a Robert Wood Johnson Foundation-funded, IHI-convened initiative, aimed to help healthy community coalitions develop readiness to transform how they approached health improvement projects. There was A LOT of data, and it was sometimes hard to know how to make sense of it. Because SCALE was a formative evaluation, we needed a process to deeply delve into of emerging results on an ongoing basis.

Hot Tip: To do so, we had periodic “Critical Moment” Sessions. The Critical Moment method uses a structured narrative approach in which the evaluator poses questions to elicit participant emotions, reactions, and beliefs on topics of interest. Reflective activities that involve stakeholders can be used contextualize the ongoing narrative of a project and ascribe meaning to experiences (e.g., What has been the most memorable moment or experience on this project? What about it made the moment so memorable? How did the moment affect you?)

We conducted these in group settings to allow all relevant stakeholders, from the implementation team, to the funder, to community representatives, the opportunity to process together. First, we sent out an “inquiry” question to participants ahead of time, which we then synthesized and grouped all responses. We shared this grouping during the session (see visual below; yup, it is crowded, there were a lot of comments!)

After allowing everyone to voice what they thought were the key moments, we held a voting process to focus discussion around 2-3 moments to explore in more depth.

This general process was so successful, we conducted over ten critical sessions over the course of the 4-year project, and saw the methods adopted by community participants to us in their own settings. The process really tapped into the spirit of community psychology putting the meaning-making into the hands of the people tasks with making program changes.

Rad Resources: Check out this online paper from MIT understand and dive into this powerful tool. http://web.mit.edu/cil/web_scripts/www/Critical%20Moments%20Methodology%20Brief%20CoLab.pdf

The American Evaluation Association is celebrating CP TIG Week with our colleagues in the Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.