Greetings AEA 365’ers. I’m Susan Staggs, an independent evaluator and community psychologist, here to talk about infusing community psychology values including respect for diversity, social justice, and prevention rather than treatment into the evaluation synthesis process.
Cool Trick: Evaluation Synthesis, as defined by Jane Davidson, is “combining evaluative ratings on multiple dimensions or components to come to overall conclusions.” It’s a simple thing to infuse these values into the synthesis process, but it’s not often done, perhaps because evaluators are overly focused on criteria such as effectiveness, efficiency, and impact. Those are wonderful criteria, but if we’re serious about valuing diversity and creating positive social change (and our stakeholders agree), we need to explicitly evaluate performance on those criteria of merit.
Hot Tip: One way to synthesize data is creating a set of rubrics to get to a final judgement. Below are examples of partial rubrics:
- Rubric 1 for determining merit on criteria:
Rubric 1: Merit Determination for Each Criteria
| Criteria | Importance Weighting | Merit Determination | |||
| Excellent | Good | Fair | Poor | ||
| Emphasis on prevention over treatment | Medium | 80% of funds devoted to prevention programming | 60% of funds devoted to prevention programming | 40% of funds devoted to prevention programming | Less than 40% of funds devoted to prevention programming |
| Explicit attention to social justice concerns | High | Free babysitting provided during 90% of education sessions | Free babysitting provided during 85% of education sessions | Free babysitting provided during 80% of education sessions | Free babysitting provided during 75% of education sessions |
| Effectiveness | High | ||||
| Efficiency | Low | ||||
- and Rubric 2 for synthesizing Rubric 1 results into a final judgment. Rubric 2 has a soft hurdle requirement, which is a requirement that must be met to obtain an exemplary final rating.
Rubric 2: Final Rating Determination
| Final rating | Performance Standard |
| Exemplary Performance | Merit determination of excellent required on all high importance criteria and good on criteria of medium and low importance |
| Soft Hurdle | Merit determination of excellent on prevention and social justice criteria |
| Proficient performance | Good merit on criteria of high or medium importance; no poor merit |
| Fair performance | Good merit on high importance criteria; all other merit determinations of good or fair; no poor merit |
Rad Resources: (on Evaluation Synthesis Methodology) The synthesis methodology chapter in Jane Davidson’s book Evaluation Methodology Basics, and her UNICEF Methodology brief on evaluative reasoning.
*Special thanks to Amy Gullickson, University of Melbourne, for introducing me to the art and science of synthesis.
The American Evaluation Association is celebrating CP TIG Week with our colleagues in the Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.