Hi! We are Brittany Dernberger and Florence Santos, and we lead efforts to measure systems-level change and impact at CARE—an international development and humanitarian NGO focused on women and girls. For the better part of a year, we led an innovative, local-to-global systems-level change evaluation of CARE’s impact on COVID-19 vaccinations. This was quite challenging, especially during the design phase, as we couldn’t find similar evaluations to draw inspiration from. The process of designing, conducting, and disseminating the evaluation created rich opportunities for organizational learning and evaluation capacity building.
The evaluation covered a vast scope: local-to-global systems change spanning local community involvement to United Nations briefings across 34 countries. We are driven to confront the challenge of holding ourselves accountable for impacting people’s lives through systems-level change, despite the inherent complexity and occasional controversy. It is unsurprising how few INGOs undertake this kind of evaluation, even as global NGOs increasingly shift towards systems-level work.
Lessons Learned
Learn-by-Doing to Refine Systems-Level Change Measurement Framework
There are two ways our evaluation of CARE’s impact on COVID-19 vaccinations increased our organizational evaluation capacity. First, we learned a lot through the process of conducting the evaluation. Like many NGOs, CARE often engages third party evaluators to ensure impartiality and independence when measuring the results of our programs. However, it’s important for us to build in-house processes and frameworks to measure CARE’s impact at scale and systems-level change, as this is a key part of our strategy. We knew internal evaluations also enhance learning and adaptive management. To strike a balance, we enlisted third party evaluators to independently assess the extent and significance of CARE’s contributions and influence. We also assembled an in-house cross-functional team and iteratively built a measurement framework that traced impact all the way to the individuals affected by the systems changes. Along with the independent assessment, we triangulated data from primary and secondary sources to increase quantitative certainty and credibility.
As a result of this learn-by-doing approach to evaluation, we have refined our systems-level change measurement framework, making it more robust. Ongoing systems-level impact evaluations are guided by this enhanced framework.
Apply Recommendations to Future Initiatives
Second, one of the big takeaways from this process was that we were not tracking the right outcomes at the beginning of CARE’s COVID-19 campaign, which made measurement on the backend difficult. Given the highly complex and constantly changing nature of systems change, we’ve identified five steps to measure impact for future large-scale initiatives and campaigns:
- Jointly identify the broader organizational metrics that are most relevant to the initiative and build campaign outcomes and metrics directly aligned to that indicator (for CARE, it’s our Vison 2030 indicators).
- Match the fundraising goals with the impact goals. If you plan to raise $1 million dollars but only end up with $100,000, that will shift impact targets.
- Instead of one standalone target number, incorporate outcomes for different scenarios, such as outcomes tied to specific levels of funding.
- Agree on specific, change-focused milestones or objectives and jointly understand the theory of change across teams and departments.
- Include “leading indicators” and incorporate mid-action reviews and/or frequent reflection points (using the leading indicators) into the project or campaign cycle, so the team is better able to determine when adjustments or pivots may be necessary.
We’re applying these lessons learned now. CARE is about to launch another campaign, and we’ve built systems-level change outcomes and impact metrics into the strategy. We’re also investing in learning by annually setting aside operational funding for impact evaluations, a practice that was implemented as a result of the utility and success of our COVID-19 evaluation.
The American Evaluation Association is hosting Organizational Learning and Evaluation Capacity Building (OL-ECB) Topical Interest Group Week. The contributions all this week to AEA365 come from our OL-ECB TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.