Hi! My name is Jaymie Lorthridge and I use a culturally responsive and equitable evaluation (CREE) approach to assess human service programs. The “C” in CREE requires attention to culture during all phases of the evaluation and when combined with the “E”, requires examination of cultural experiences by assessing outcomes related to equality, equity, and ideally social justice (EESJ). A recent project offered an opportunity to examine EESJ components by using different subsamples or comparison groups. The internal conversation underscored the importance of decisions about who will be compared, according to which identity characteristics and demographics (such as race, gender, age), and why; and we discussed the importance of having guides to inform decisions. There are issues with using identity characteristics as cultural proxies. That’s another blog post!
Lesson Learned
Our team used the clients’ EESJ definitions and frameworks to guide decisions about whom to compare to and to identify how data displays could either create opportunities for EESJ dialogue or restrict the data story.
Hot Tip
Make sure project-specific EESJ definitions are in place. Almost everyone is trying to advance equitable or socially just outcomes. Programs are often designed to affect equality. Co-developing and socializing EESJ definitions ensures that program implementers, funders, and the persons whose lived expertise is hopefully informing the creation or expansion of services, understand each other’s expectations and desired outcomes. If EESJ definitions are not available when your partnership begins, definitions can be developed during your theory of change or logic model work or when you operationalize constructs in your research questions.
Rad Resource
Tony Ruth’s graphic depicts inequality and EESJ. The brief captions are a useful starting point for EESJ definitions and identification of context-specific factors. No visual is perfect. The linked resource offers a critique of the images and when I use the graphic, I acknowledge the implicit judgements in each image. I push the group I’m working with to be honest about what is realistic in the given service environment and what is ideal.
Lessons Learned
Each evaluation phase presents an opportunity to consider how comparisons can provide EESJ insights.
- Comparisons get a lot of attention during the design and analysis phases.
- Comparisons should get just as much attention during the dissemination phase. Visualized comparisons can conflate EESJ definitions. Often “equity” visualizations show included racial groups in comparison to whites, conflating equality with equity.
Hot Tip
- Emphasize equality by comparing across groups with varying identities (e.g., racial groups compared to each other).
- Emphasize equity by using within-group comparisons, over time and within pertinent context (i.e., examine if one group changed when compared to past measurement). Within group comparisons can also focus on participant subgroups whose data might be withheld due to its smaller sample size.
- Emphasize social justice by using within group comparisons, over time, within pertinent context and alongside indicators of systemic oppression (e.g., race-specific, pretrial bail amounts for the same charge) and reform (e.g., timing for implementation of race-blind bail assignment represented in a visual depicting outcomes).
- Keep validity threats in mind for all comparisons.
- Frame comparisons within context. Pertinent context includes, but is not limited to, service quality, at minimum according to service recipients; the cultural appropriateness of services; and community conditions inclusive of marginalization indicators such as historical blocks to accumulating wealth (e.g., redlining).
Rad Resources
- The Urban League’s Equality Index provides an excellent example of how to frame discussions about equality.
- Equity gap scores can be added to within group comparisons to establish group-specific benchmarks, though the presence of scores might invite cross-group comparisons.
Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.