I’m Linda Stern, Director of Evaluation & Learning at the National Democratic Institute. I also lead a Research, Evaluation & Learning (REL) initiative with seven other democracy, human rights and labor (DRL) organizations under the Fundamental Freedoms Fund (FFF). Sponsored by the US Department of State, the objective of our consortium initiative is to build a more meaningful evidence base for the Fund’s short-term programs aimed at preserving and protecting democratic space threatened by endogenous factors such as authoritarian consolidation; and exogenous factors such as environmental disasters, refugee flows, economic downturns and pandemics. In partnership with the EGAP and the Democratic Erosion academic networks, the consortium members have grappled with how to evaluate democracies in crisis. We offer our emerging lessons learned below:
- Use a systems framework, not a results framework: Short-term, rapid response interventions in crisis contexts are inherently difficult to monitor and evaluate. During crises, the context is in flux and traditional results frameworks — premised on logical hierarchies and static contexts — are often too rigid for the rapidly changing environments that overturn our programs’ critical assumptions. Exogenous shocks to the democratic system, like the Global COVID19 pandemic, not only create health and economic crises, but open up opportunities for authoritarian consolidation of power. Taking a systems approach helps to nest short-term interventions within a web of interdependent, interacting and interconnected relationships within the political ecosystem.
- Evaluate changes in democratic accountability: Democratic systems are premised on a web of interconnected accountabilities, both defacto and dejure, meant to create a power equilibrium between citizens and their governments. Multiple factors can erode these lines of democratic accountability and redistribute power within the political system. A number of consortium members have used a vertical, horizontal and diagonal accountabilities framework to identify clear domains of change within the democratic system, allowing them to compare accountability in closing space in Hungary, against opening space in The Gambia.
- Use ex post evaluations to understand systems change over time: The ripple effect of short-term interventions cannot be easily observed within a project lifecycle. Ex post evaluations include the variable of time to retrospectively map potential causal pathways, changes in political relationships and political networks, as well as the sustainability of program effects. Some consortium members are using social network analysis to evaluate the strength and sustainability of an advocacy coalition in Cambodia. Collectively, these ex post case studies are generating new consortium hypotheses for our short-term programming.
- Use cross-national panel data to identify patterns: Democratic crises are brought about by both shocks to the system, and the historical antecedents that increase risks for democratic crises. The consortium’s work with academic partners has underscored the importance of using cross-national panel data to identify patterns of democratic erosion and autocracy over a longer time horizon (DEED); or to diagnose the complexities of a democratic system, using multidimensional indices (V-Dem). Both these datasets help to create comparative case study frameworks for our ex post evaluations, as well as inform the design of future short-term interventions for democracies in crisis.
The American Evaluation Association is celebrating DemTIG Week with our colleagues in the Democracy, Human Rights, and Governance TIG. All of the blog contributions this week come from our DemTIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.