My name is Scott Chaplowe and I’ve been a member of the AEA since 2002, currently serve on AEA’s International Working Group, and I am the Coordinator of the Transformational Evaluation Working Group in the International Evaluation Academy, (IEA).
I often get asked what is transformational evaluation? It is not a specific approach or methodology like Developmental Evaluation or Realist Evaluation. Instead, I use transformational evaluation to refer to evaluation that supports transformational learning and change to respond to today’s formidable challenges. The next question I get is how? My answer is that evaluation needs to transform from within if it is to contribute to the urgent transformations needed in the world. I identify four interrelated “boxes” that confine evaluation’s transformational potential, which I will now briefly summarize here…
The Project Box
We need evaluation that supports accountability to the planet (and its inhabitants) rather than discrete projects/programs and those who fund them. The project fixation reflects a preoccupation with conceptual (logic) models or frameworks to identify casual linkages for pre-determined and their KPIs (key performance indicators). This makes measurement more doable, which is good for accountability, but it leads to reductionist planning and analysis that reinforces siloed rather than systems thinking. The result is that evaluation fixates on an intervention’s intended results, overlooking or downplaying other critical considerations (see Figure 2).
Figure 2: Linear Intervention Design
The Temporal Box
A project/program is designed with a given timeline, often dictated by the funding cycles of the donor. This preconceived timeframe and payment schedule is based on how the design model predicts change over time. It reflects a mechanistic casual model that evokes static predictability, order, and timing. However, complex systems do not behave according to project budget cycles. There is a myriad of emergent, intervening variables that can affect the intended intervention logic. For instance, consider how the 2008 Great Recession and the COVID-19 pandemic have significantly impacted the timing (and budgets) of programs and projects.
The Quantitative Box
Whereas it is often asserted that, “What gets measured gets done,” another witticism reminds us that, “Not everything that can be counted counts, and not everything that counts can be counted.” Reality is not a binary concept that can be calculated, nor wrapped up into neat, quantitative boxes with KPIs that measure impact achieved. The same attributes that make systems complex make interventions within hard to measure, and a “tyranny of metrics” engenders excessive bureaucratization and “proceduraliztion” that handicaps flexibility and innovation which is crucial for transformation.
Another temporal concern is that conventional summative (final) evaluations typically used to assess interventions are commissioned with an endpoint or exit strategy in mind, which narrows assessment to the short- and mid-term outcomes. This shortsightedness neglects the evaluation of future unintended consequences (externalities) on the human and natural ecosystem.
The Accountability Box
The evaluation marketplace often consigns evaluation to a descriptive, tick-box, accounting exercise fixated on whether something is done “right” versus whether the “right” thing is being done in the first place. Historian Jerry Muller, author of The Tyranny of Metrics, observes, “The most characteristic feature of metric fixation is the aspiration to replace judgement based on experience with standardized measurement.” Michael Scriven warns of such “valuephobia,” which undermines the very essence of evaluation to “determine merit, worth, value, or significance.”
Hot Tip & Rad Resource
Every problem carries the kernel of and points to its solution, which points to complex systems analysis as a means to “breakout” of these boxes and nudge evaluation towards the inner transformation required for it to contribute to transformational change. Read more on this topic in the freely accessible Evaluating Outside the Box: Evaluation’s Transformational Potential (Chaplowe and Hejnowicz. 2021. Social Innovations Journal).
The American Evaluation Association is hosting International Evaluation Academy (IEA) Week. The contributions to AEA365 this week are all related to this theme. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.