This week’s posts highlight reflections from the Global Evaluation Initiative (GEI), a global network of organizations and experts working together to support the strengthening of monitoring, evaluation, and the use of evidence in developing countries. GEI uses an integrated systems-based approach and works closely with governments, evaluation professionals, and other stakeholders on efforts that are country-owned and aligned with local needs and perspectives.
-Liz DiLuzio, Lead Curator, AEA365
Hi, we are Lycia Lima (Deputy Director), Gabriela Lacerda (Executive Manager), and Lorena Figueiredo (Researcher) of the Center for Learning on Evaluation and Results for Lusophone Africa and Brazil (CLEAR-LAB), which is based at the School of Economics of Getulio Vargas Foundation and is an Implementing Partner of the Global Evaluation Initiative (GEI).
We are going to share some Lessons Learned from our research on evaluation practices and systems in Brazilian subnational governments. If you are carrying out policy research across different local institutional contexts or if you are interested in mixed methods research, then this blog is for you.
We sought to assess the evaluation capacity of Brazilian states and municipalities. We were inspired by the methodology of GEI’s Monitoring and Evaluation Systems Analysis (MESA) tool. MESA is a diagnostic tool that can be used to gather, structure, and analyze information to form a comprehensive picture of a national monitoring and evaluation (M&E) ecosystem.
We’d like to share some of the lessons we learned from this work:
MESA is a flexible instrument: As a diagnostic tool, MESA can be used to structure the research design and to define research questions. We began our research with the basic questions for “Evaluation Systems” described in the MESA Guidance Note. The questions look at a government’s capacity to manage and undertake evaluations, coordinate an evaluation system, and enter partnerships with academia, civil society, and NGOs. The questions also explore what mechanisms exist to ensure that governments deploy evaluation results.
Comparing requires a common backdrop: Once our main research questions were defined, our next task was to adapt MESA to the local level. To be able to compare different institutional contexts, we used the same questionnaire for all states and cities. We needed to capture the variability of institutional contexts across Brazilian states and cities while generating comparable data. For that, we adapted the OECD Survey on Policy Evaluation, adjusting where necessary to better reflect the local context. The Survey functioned to translate the MESA open-ended questions into closed-ended questions that would enable quantitative analysis.
We collected answers from 26 of the 27 Brazilian states, including the Federal District, and from 15 municipalities, focusing on capital cities and cities with a population of over 500,000. The questionnaire was directed to civil servants in departments of planning or in the cabinet – the so-called “center of government.”
Governments are dynamic: Paying attention to the electoral cycle and the budget process helped us to be aware of institutional changes and priority shifts. Interviews were used to clarify points and get updates about the institutional context since the research process overlapped with the change of electoral mandates in states.
The results of the research will be published in a report. It will be the first comprehensive study of institutional practices of policy evaluation at the subnational level in Brazil.
Moreover, the research process was a way to get to know local policymakers and their institutions better. Through the survey and interviews, we fostered connections. These connections led us to develop an online synchronous course on policy evaluation with 90 participants. In the course, policymakers working in the field of M&E exchanged experiences and learned from each other. Therefore, one final lesson learned is that the research process contributed to the early development of a network of evaluation experts in Brazil.
The American Evaluation Association is celebrating Global Evaluation Initiative (GEI) Week. The contributions all this week to aea365 come from GEI members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.