CP TIG Week: Cool Trick for Infusing Community Psychology Values into the Evaluation Synthesis Process by Susan Staggs

Greetings AEA 365’ers. I’m Susan Staggs, an independent evaluator and community psychologist, here to talk about infusing community psychology values including respect for diversity, social justice, and prevention rather than treatment into the evaluation synthesis process.

Cool Trick: Evaluation Synthesis, as defined by Jane Davidson, is “combining evaluative ratings on multiple dimensions or components to come to overall conclusions.” It’s a simple thing to infuse these values into the synthesis process, but it’s not often done, perhaps because evaluators are overly focused on criteria such as effectiveness, efficiency, and impact. Those are wonderful criteria, but if we’re serious about valuing diversity and creating positive social change (and our stakeholders agree), we need to explicitly evaluate performance on those criteria of merit.

Hot Tip: One way to synthesize data is creating a set of rubrics to get to a final judgement. Below are examples of partial rubrics:

  • Rubric 1 for determining merit on criteria:

Rubric 1: Merit Determination for Each Criteria

Criteria Importance Weighting Merit Determination
Excellent Good Fair Poor
Emphasis on prevention over treatment Medium 80% of funds devoted to prevention programming 60% of funds devoted to prevention programming 40% of funds devoted to prevention programming Less than 40% of funds devoted to prevention programming
Explicit attention to social justice concerns High Free babysitting provided during 90% of education sessions Free babysitting provided during 85% of education sessions Free babysitting provided during 80% of education sessions Free babysitting provided during 75% of education sessions
Effectiveness High
Efficiency Low

 

  • and Rubric 2 for synthesizing Rubric 1 results into a final judgment. Rubric 2 has a soft hurdle requirement, which is a requirement that must be met to obtain an exemplary final rating.

 

Rubric 2: Final Rating Determination

Final rating Performance Standard
Exemplary Performance Merit determination of excellent required on all high importance criteria and good on criteria of medium and low importance
Soft Hurdle Merit determination of excellent on prevention and social justice criteria
Proficient performance Good merit on criteria of high or medium importance; no poor merit
Fair performance Good merit on high importance criteria; all other merit determinations of good or fair; no poor merit

 

Rad Resources: (on Evaluation Synthesis Methodology) The synthesis methodology chapter in Jane Davidson’s book Evaluation Methodology Basics, and her UNICEF Methodology brief on evaluative reasoning.

 

*Special thanks to Amy Gullickson, University of Melbourne, for introducing me to the art and science of synthesis.

The American Evaluation Association is celebrating CP TIG Week with our colleagues in the Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.