Hi there! We’re Ransford and Cris, an evaluation team with the University of Florida and Performance Hypothesis.
We recently wrapped up a rapid review looking at how participatory approaches have been used to evaluate arts-based mental health interventions. The work grew out of Thriving Together Atlanta, a Bloomberg Philanthropies-supported initiative using public art to address systemic racism as a public health crisis. While building evaluation tools for that project, we realized we needed a clearer picture of what participatory evaluation looks like in practice across this field. So, we went looking, and what we found has real implications for evaluators thinking about how to do this work with integrity.
Lesson Learned: “Participatory” can mean a lot of things, and the differences matter.
One thing that profoundly stuck with us was the realisation that although evaluators and academics were involved in 18 of the 22 studies, community members showed up in fewer than half. And participation tended to cluster early, during design and data collection, then drop off sharply during analysis, interpretation, and dissemination. We interpreted this as inviting people to the table but making the final decisions without them is consultation, not participation. For evaluators committed to culturally responsive evaluation, this distinction is critical. For example, we imagined how good it would be to have community members handle dissemination of findings instead of researchers and academics. This would mean that most of the first authors of papers are community members, or community members are given the opportunity to present findings of the evaluation at conferences, or community meetings. This is where local participation and ownership is truly demonstrated.
Hot Tip: Know where culturally responsive evaluation and PAR overlap.
Culturally responsive evaluation asks us to honor cultural context, examine power, and center the perspectives of those most affected. While participatory action research shares many of those commitments it adds an explicit action orientation highlighting that the community does not merely inform the evaluation but co-produces knowledge and drives change. In our review, the studies that scored highest on participatory rigor were the ones that carried community involvement all the way through to action planning and dissemination.
Rad Resource: The community-partnered critical appraisal tool is worth adding to your toolkit.
We used it to assess how well each study followed through on its participatory commitments, and it surfaced gaps we might have missed otherwise, especially around power-sharing, community resourcing, and whether findings were disseminated in ways communities could use. Community resourcing was one thing that stood out most for us especially regarding studies that worked in marginalized and deprived communities. Resources like stipends and funding for the community organisations would have gone a long way to encourage participation from community members as most studies saw a lot of dropouts during their evaluation processes and we suspect that this could be a result.
Lesson Learned: Be honest about what your evaluation can and cannot claim.
Nineteen of 22 studies reported positive outcomes. That sounds great until you consider what might be missing from the published record. Evaluators working in participatory spaces should resist the pressure to present only success stories. Reporting mixed or unexpected findings is not a failure. It is how the field learns. And when the creative activity doubles as both the intervention and the evaluation method, as we saw in several studies, being transparent about that overlap is essential for credibility.
Lastly, we would like to thank our co-authors: Marcel Foster, Yaning Wu, Olaitan Komolafe, Rachael Casey, Jane Morgan-Daniel for their amazing work on this review.
The American Evaluation Association is hosting Arts, Culture, and Museums (ACM) TIG Week. The contributions all week come from ACA TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.