Thank you for subscribing to aea365! AEA is aware of the issue impacting email receipt of aea365 blog posts. Our RSS Feed provider has announced a reduction in maintenance which has unfortunately resulted in sporadic email receipt of daily posts. AEA is transitioning to a new provider and intend to roll out a new, improved service in early September 2022! Until then, please continue to regularly check back to aea365 for daily tips, tricks, and resources for evaluators by evaluators. You may also add AEA365 blog’s feed URL – https://aea365.org/blog/feed/ – to your favorite RSS! Stay tuned for additional details! For questions, please contact AEA at info@eval.org.
Hello, we are Laura Adams, a MEL Director at Pact, and Sarya Sok, Acting Team Lead for Applied Learning and Evaluation at the State Department’s Bureau of Democracy, Human Rights, and Labor. We are fans of solving wicked MEL challenges in democracy, rights, and governance programs (aka pulling for the DRG underdogs with our MEL muscles).
For years, we have been asking ourselves: how can evaluators approach the problem of documenting the long-term contributions of program interventions in restrictive contexts? Long-term contributions are difficult to identify under ideal circumstances, but in restrictive contexts, documenting DRG contributions can also put people in danger. Yet, we persist (with care!) and a method called process tracing has helped us get around the limitations of traditional program evaluation methods. Process tracing (PT) is a rigorous, retrospective, within-case method for determining causal mechanisms. It requires deep contextual knowledge, access to diverse sources of evidence, and the ability to clearly spell out and document the connection between hypothesized potential causes and the outcome of interest.
The outcome we’re studying is the growth of civic mobilization in authoritarian contexts. Democracy and rights activists in such contexts often just hold the line for decades. As MEL practitioners, we strive to understand whether our partnerships with activists laid the groundwork that, years later, contributed to social change.
We have traced the precursors of civic mobilization in four recent cases (forthcoming in fall, 2022), looking for evidence that the kind of programs we do in the DRG sector made a contribution. In the Sudanese case, the evidence makes us confident that particular DRG interventions did make an important causal contribution to the 2019 revolution. In our Belarus 2020 case, however, we learned that the contribution of DRG partners was small in comparison to that of a savvy new generation of political entrepreneurs. In Belarus and Vietnam, journalists, lawyers, and civic organizations played important but non-causal roles. In our Ethiopia case, we were not able to find any high-value evidence that DRG interventions made a contribution until the mass movement had already succeeded in forcing a new political settlement.
Tracing causality in these cases was humbling, but it is healthy to decenter “the sector” from the story of democratization. PT allows us to think of the factors we discovered as a menu of programming options linked to particular conditions. For example, the Ethiopia PT helps us think more creatively about how future program designs can take into account the role of diasporas in supporting civic mobilizations in restrictive contexts.
Lessons Learned
- Process tracing should be done by country experts who have access to diverse sources.
- The outcome needs to be clearly and narrowly defined. “Why did this mobilization grow in size at this time?” rather than “why did people protest?”
- Competing hypotheses about “why at this time?” need to be matched with evidence and evaluated, though we found that PT’s method to systematically weigh evidence was not helpful in examining events of this complexity.
- In two of our cases, high-value evidence came from activists inside the country who might be in danger if their identity was revealed. We tried to work with scholar/activist pairs both inside and outside the country, and we enforced strict digital and physical security protocols, even when the researchers did not think they needed them!
Hot Tips
- People with journalism training make better process tracers than evaluators and academics. PT requires good investigative and interview skills, and someone who can see both the trees and the forest.
- Plan to spend a significant amount of your time spelling out the sequence of events and the role of different actors before you jump into weighing the evidence for and against different hypotheses.
- Whether or not you use PT, solving wicked MEL problems requires humility, outstanding interviewing skills, and analytical chops.
The American Evaluation Association is hosting Democracy, Human Rights & Governance TIG Week with our colleagues in the Democracy, Human Rights & Governance Topical Interest Group. The contributions all this week to aea365 come from our DRG TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.