Hi! We’re Thomaz Chianca and Jane Davidson. We have been collaborating over the past several years to help organizations use evaluation to drive and learn from systems transformation that addresses both equity and environmental sustainability.
It is challenging to evaluate change efforts that have closed out several years earlier – especially when the evidence is patchy and thin. We are here to share some useful lessons from an RIE we conducted for Save the Children in 2019, assessing the lasting effects of their first child sponsorship-funded community development effort in Ethiopia, which ran from 2002 to 2010 in a region called Woliso.
What exactly is a retrospective impact evaluation (RIE)? Also referred to as ex-post evaluation or post-project evaluation, RIE is used to assess how well outcomes, impacts, infrastructure, and/or services have been sustained long after a change effort has been wrapped up. Many evaluations assess whether these things seem likely to last; but an RIE checks to see whether they did in fact last. A good RIE typically also identifies what factors helped or hindered how long things lasted.
Pro Tip:
Don’t underestimate the challenge of finding out what was implemented! When we conducted this RIE for Save the Children, it had been 17 years since they had first arrived in the impact area. Records from so far back in time were patchy and memories somewhat foggy, so we needed to do a lot of detective work to fill the gaps!
Lesson Learned:
A retrospective impact evaluation isn’t just a “looking in the rear-view mirror” exercise! Most clients want to understand how to make their future impact in human and natural systems more long-lasting and sustainable. Two particularly useful contributions are: (i) identifying the factors that made things last a long time with little maintenance and not harm the environment, and (ii) highlighting the most important trade-offs (e.g., training more educators than initially needed costs more up front but helps future-proof the change for growing populations and teacher turnover; concrete buildings last longer but are terrible for the environment).
Lesson Learned:
When stakeholders suggest multiple possible secondary sources of data, check them out but don’t get too excited, and make sure you have a Plan B. Although many datasets look promising, most will be at too broad a level of analysis to be usable (e.g., health statistics for an entire region much larger than the impact area) or won’t have the outcomes you need.
Pro Tip:
Use evaluation rubrics as a rich, nuanced way to capture the lived experiences of children, families, and community members in a way that embodies locally relevant values.
Rad Resource:
If you’re new to evaluation rubrics methodology, check out Jane’s webpage for a brief explanation: What are rubrics?
Pro Tip:
Work with your client’s Communications team to ensure that any explanations of your findings are accurate. This is particularly important for Value for Investment analyses, which are easy to misstate. Check out the video explainers we worked on with Save the Children to summarize the main findings from the evaluation.
Rad Resource:
One of the best ways to learn how to do RIE is to get coaching from a colleague with experience in that area. If that’s not an option, here’s another great resource. To help our evaluator colleagues see how various concepts, approaches, and methodologies can be combined in the real world to create an useful, practical RIE, we created a case-based RIE e-course that uses the above-mentioned Ethiopia evaluation (see the outline below).
The American Evaluation Association is hosting International and Cross-Cultural (ICCE) TIG Week with our colleagues in the International and Cross-Cultural Topical Interest Group. The contributions all this week to AEA365 come from our ICCE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.