Assessing the Feasibility of Conducting a Retrospective Impact Evaluation by Larry Dershem, Ashley Bishop, and Brad Kerner

Greetings! We are Larry Dershem (Senior Advisor, Research & Evaluation), Ashley Bishop (Monitoring, Evaluation and Learning Advisor), and Brad Kerner (Senior Director, Sponsorship Program) working for Save the Children (SC), which is an international charitable relief and development organization that seeks to ensure children survive, learn, and are protected internationally and in the US.

A retrospective impact evaluation (RIE) is an ex post evaluation of an evaluand to assess its value, worth, and merit, with a special focus on examining sustainability of intended results as well as unintended impacts. However, due to resource constraints, international development donors and organizations cannot afford to conduct many RIEs, limiting our ability to truly understand longer-term outcomes and impacts after the closure of a program. Save the Children’s sponsorship programs are currently investigating the feasibility of conducting RIEs in order to optimize learning from scarce resources.

With invaluable professional assistance from E. Jane Davidson (RealEvaluation) and Thomaz Chianca (independent consultant), we developed a RIE Evaluability Scoping Guide to assess the feasibility of conducting an RIE. The RIE Scoping Guide assesses 24 issues categorized into the following four dimensions: 1) Internal Stakeholder Support, 2) External Stakeholder Support, 3) Availability of Evidence & Documentation, and 4) Context. After program staff review, discuss and score each of the 24 issues, a tally of the scores indicate which of the four feasibility categories best describes the feasibility of a RIE.

RIE Feasibility Categories

To date, five sponsorship programs implemented for 10-years, and ended 5 to 9 year ago in Bangladesh, Nepal, and Ethiopia, have been assessed. All five programs are either “Adequately Evaluation Ready” or “Fully Evaluation Ready”; therefore, in the next year SC plans to commission at least one RIE.

Lesson Learned:

• To avoid confusion, be clear with program staff that an evaluability assessment of a program is NOT an evaluation of a program.
• For program staff to clearly asses the evaluability issues under the 4 dimensions, each issue should have a short description and set of questions to be answered.

Hot Tips:

• Having specific dimensions and issues that are critical for a RIE to be feasible after a program has ended allows staff to incorporate these issues into the program from the beginning.
• SC has established an optimal window of 5-10 years after completion of a program to conduct a RIE which allows for longer-term impacts to occur but not too long to limit the number of confounding factors.

Rad Resources:

• Davies, Rick. (2013). Planning Evaluability Assessments: A Synthesis of the Literature with UK Department for International Development.
• Dunn, Elizabeth (2008), “Planning for cost effective evaluation with evaluability assessment.” Impact Assessment Primer Series, Publication #6, United States Agency for International  Development (USAID).
• Khatiwada, Lila Kumar. (2017). Implementing a Post-Project Sustainability Study (PSS) of a Development Project: Lessons Learned from Indonesia. Reconsidering Development. Vol. 5 No 1.
• Peersman, Greet, Irene Guijt & Tiina Pasanen. (2015). Evaluability Assessment for Impact A Methods Lab Publication. ODI.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.