I’m Carlos Romero from Apex in Albuquerque. We are systems evaluators working with health, education, and social programs that address the wicked problems of inequity. I aspire to Equitable Evaluation (EE) which the Center for Evaluation Innovation defines as “using evaluation as a tool for advancing equity and involves diversity of teams, cultural appropriateness and validity of methods, revealing drivers of inequity, empowering those more affected to shape and own how evaluation happens.” I’m all in on this vision and I’d like to share some thinking I’ve been doing about the practice and business of EE.
First, I believe the EE movement is lacking an effective mission – one that is:
- Actionable – it tells you what to do;
- Clear and simple – unambiguous and easy;
- Repeatable – but iterative and emergent; and
- Universal – everyone can do it together.
Apex follows a model of systems learning as our mission. It’s a practice that scrutinizes numbers and elevates perspectives, informed by:
(Rad Resources)
 | Counting. The case for scrutiny: “Every number is born of subjective judgments, points of view, and cultural assumptions. Numbers are filled with bias through and through, because that’s what categories do.” |
 | Decolonizing Wealth. Among many insights relevant to EE, why positionality matters: “Those most excluded and exploited by today’s broken system possess exactly the perspective and wisdom needed to fix it.” |
Changing hearts and minds is important, but we also need to get to work. So, let’s talk business. Supply and demand. The missional attributes above are also qualities of a good game – and every good game involves an economy that tells players what’s important and incentivizes whatever outcome the game designers have devised. I’m convinced that the current economy of evaluation is incongruent with the outcome of learning. Here’s why:
- Learning requires authenticity and leads to noticeable change.
- Every system is getting the results it was designed to get…for someone.
- Power is the ability to change the rules of the game…or not.
I hear people inside philanthropy and the “evaluation industrial complex” shrug off the lack of authenticity and accountability around EE, saying things like “everyone knows the biggest sponsors are the biggest violators,” and “that’s just the way it is – what are you going to do?” Now, the organizations that already dominate the industry announce they are getting in the equity game. Starting with “workforce diversity,” which is great but a misunderstanding of positionality, they promise to come up with a “range of opportunities” and “facilitate dialogue” around “inclusive practices.” Noisy jargon. Call me cynical, but based on the history of the field, the notion that these institutions will transform themselves without any specificity or accountability for authenticity and change…it’s like Big Tobacco leading the charge on smoking cessation.
To be clear, my indictment is of the system of evaluation supply and demand and does not impugn the integrity of the people and organizations doing meaningful work. It’s just the game we’ve all been given to play – and we shouldn’t be surprised when the status quo protects itself. It’s the nature of the system and power dynamics associated with wicked problems – and EE lives inside the super wicked problem of philanthropy.How do we disrupt these forces? Flipping the business model of methods would help. More systemically, we should look closely at the certification processes for organic foods and green construction. We learn from analogy, and these are two models of transformation that resulted in healthier, more sustainable industries. One way or another, if Equitable Evaluation is to mean anything, there must be standards with accountability that permeate its practice and business – supply and demand.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.