My name is John Gargani. I’m a former AEA President, and I have a deep interest in how impact investors, social entrepreneurs, and other new actors measure their impact.
At Evaluation 2022, I introduced the idea of Zones of Greater Scrutiny, or ZOGS, to help evaluators defend against impact washing. In this post, I describe how to incorporate ZOGs in evaluative rubrics.
The Problem of Impact Washing
New actors seem obsessed with creating positive impact. I believe most are sincere, and those I work with are dedicated to reversing climate change, promoting social justice, and addressing many other critical problems. They also want to make money. Importantly, they make more money when they convince others of their impact.
Consider a social enterprise that helps unemployed young people find meaningful jobs. It can charge clients more, attract new investment, and scale faster if clients, investors, and partners believe it has a positive impact.
This brings us to the problem of impact washing, which occurs when organizations overstate their impacts. Sometimes impact washing is willful, at other times wishful, but it is always a problem.
A Possible Solution: Evaluative Rubrics with Zones of Greater Scrutiny (ZOGS)
At their simplest, evaluative rubrics have two components—criteria and standards. Criteria identify what stakeholders consider salient when judging merit, worth, or significance. Intended impacts—like increasing the number of young people with meaningful jobs—are often used as criteria. Standards establish cutoffs for success—for example, at least 20% of enrolled jobseekers find work because of the program.
However, some impact claims may seem too good to be true. If a social enterprise claimed that 100% of participants found jobs because of its efforts, it would more than meet our standard for success. It would also be hard to believe.
The principle behind ZOGS is known as ECREE, which stands for extraordinary claims require extraordinary evidence. It’s a principal associated with the late astronomer Carl Sagan. ECREE suggests that some claims are more difficult to accept than others because they contradict a body of past results, violate common sense, or challenge prior beliefs.
Let’s say, for example, that stakeholders agree that 70% finding jobs is the smallest impact that strains credulity. Then the ZOGS would extend from 70% to 100%. Claims falling within this range would require more credible evidence than claims falling below it.
An example:
Stakeholders should define what makes evidence more credible. It’s the evaluator’s job to scrutinize the evidence to answer the following question—Is the unusually large impact the result of an innovation, or did the organization measure its impact poorly?
Using ZOGS in evaluative rubrics can help ensure that claims are not accepted without sufficient evidence. It does not suggest what is true or false, but what level of evidence is necessary to accept claims. This can help maintain the integrity of the evaluation process and identify potential impact washing. Because organizations know in advance that extraordinary claims about impact will trigger greater scrutiny, ZOGS may also help prevent impact washing from happening in the first place.
Rad Resources: More on evaluative rubrics.
More of my writing on innovation and scaling impact.
The American Evaluation Association is hosting Social Impact Measurement TIG Week with our colleagues in the Social Impact Measurement Topical Interest Group. The contributions all this week to AEA365 come from our SIM TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Thank you so much for the very interesting blogpost and concept of ZOGS. When credible criteria are defined based on benchmarking, it will be a very powerful tool to ensure accountability of impact measurement by impact investors and social entrepreneurs.