Greetings, I am Holta Trandafili, a researcher and evaluator captivated by sustainability theories and the sustainment of results. I believe that a thoughtful, systematic inquiry of what happens after an intervention ends adds value to what we know about sustainability. Since 2015 I have co-led ten post-program evaluations (also known as ex-posts) in Uganda, Kenya, Sri Lanka, India, Myanmar, and Bolivia. Their findings point to questions and issues of theory, measurement, and sustainability expectations relevant to any program:
- To what do we compare results to judge success? Is it that 60% of community groups or water points being operational three years after closeout a good result? Should it be 87% or 90%? Why? Should we use the end-line as the measuring yardstick, especially as contexts change? Whose view of success counts?
- How long should we expect results, or community groups left behind, or activities to continue post-program? Two or ten years or Forever? Why?
- Is going back once enough to make a judgment on sustainability? What would we find if we went back in 2020 where we evaluated ex-post in 2015 or even 2019?
Lessons Learned: Here are my reflections and resources on sustainability:
To the enthusiastic evaluators ready to start ex-posts
- Lesson learned: Organizations often carry out ex-posts for accountability. However, greater wealth lays in learning. Make learning part of your evaluation objectives. It took my organization 5 years from the first ex-post to have more open conversations and share our sustainability learning on what to improve: how we design, transition, and measure programs’ impact. Now we are genuinely more accountable.
- Get involved: Don’t lose heart if your first ex-posts prove difficult to conduct or have mixed results or unearth new questions and insights on sustainability. You are not alone. Find another evaluator that has gone through an ex-post experience and ask them to write a blog, present at a conference, write guidelines, attend a course, or merely meet to vent and dream.
To those already fighting to mainstream ex-post measurement in their organizations or their clients
Hot Tips:
- Mainstreaming ex-post evaluations is commendable for any institution. In this process we should start making the case to pilot longitudinal ex-post measurements (i.e., going back not once but several points in time). We can truly unpack the issues of temporality and longevity for sustainment of results. See JICA’s example on ex-post monitoring.
- Invest in theory-driven evaluations like Realist Evaluation to unpack the hidden mechanisms behind which different types of outcomes are sustained, asking: among whom, in what contexts, how, why?
Rad Resources:
- Read insightful blogs and resources from Valuing Voices, an Advocacy Research and Evaluation Network dedicated to providing Sustained and Emerging Impacts Evaluations
- Marvel at JICA’s ex-post monitoring, returning twice post-exit
- A discussion note on Ex-posts from USAID illustrating donors interest and thinking around ex-posts
- A webinar on Sustainability Ready: What it takes to Measure and Support Lasting Change (by Isabella Jean, Holta Trandafili and Jindra Cekan/ova)
This week, AEA365 is celebrating Ex-post Eval Week during which blog authors share lessons from project exits and ex-post evaluations. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Thanks for this. Your points about when to conduct an ex post, how many times, how often/ at what intervals and what to compare to is significant. In most liberatory endeavors, the immediate sign of a successful intervention or strategic irritant is backlash. In other cases, what may may look like success soon reverts to pre-intervention states–as in US school and residential desegregation efforts.
Ways of knowing rooted in the European Enlightenment, modernism, and capitalism would have us believe in “forward” movement (as opposed to “backward” societies and peoples, or folks “left behind”) and linear progress/ progression from changes in knowledge to attitudes to behaviors to conditions, as if individual agency can control individuals’ destiny without corresponding structurally mediated changes in power dynamics and movement of resources. Oppressive forces also consistently co-opt any effective changes such that indications of success may look like “progress” but have in fact lost their edge and serve as mere shells without substance (AKA buzz words like diversity, equity, inclusion).
Transformative change is not linear. It is cyclical and exponential. People on the ground tend to know what to expect and when, as they’ve been there before, during, and after projects have come and gone for decades now. It’s the aid workers and evaluators that “move on” to their next project.