Greetings! I am María Fernanda Rodrigo and I am a Senior Associate evaluator at the Office of Evaluation and Oversight at the Inter-American Development Bank, where my research focuses on productive sectors.
Have you ever found zero effects in your impact evaluation? After incorporating many specifications, the data is clear: your program has no effect whatsoever. You start thinking: what went wrong in the surveys? Should I try to work other ideas? Maybe I can find lessons learned from this failure…
This is the case from a recent evaluation of an agricultural program in Paraguay that promoted environmentally sustainable practices and technologies in the most common crops of the region. The analysis was done five years after the program was implemented, and preliminary analysis suggested that, although practices were still being implemented, there was no impact on productivity nor income. How disappointing is this in a time when the world is trying to be more sustainable?
However, “why were these farmers still using the technologies from the program if we were not detecting any -tangible- benefit out of it?”
What were we overlooking, perhaps spillover effects? Since the technologies were not expensive, it was likely that non-direct beneficiaries also adopted these practices after realizing the benefits from direct beneficiaries. But we couldn´t estimate them with no information from individual’s networks, and those surveys are expensive. Back to our data, finally some good luck: our surveys had included geospatial information of each farmer.
So, we assumed that control farmers at a certain distance or more from a direct beneficiary were clean controls. Then we estimated the effect of the program by comparing productivity outcomes from clean controls and direct beneficiaries. Our findings indicate now that the program increased productivity by at least 45%, this effect being higher for those with more time in the program.
The technologies promoted by the program are compatible with the environment and natural resources. No forested lands were dismantled, nor natural forests replaced with intensive crops during its implementation. This paper (forthcoming) points out how the promotion of environmental technologies can be successful among small producers when incentives are aligned. Not only they must be economically viable, but also the producer needs to be involved in the technology choice, and the government aware of the adequacy of inputs and technical assistance.
Lesson Learned:
One shouldn´t wait to have a zero-impact program to analyze possible spillover effects. Many programs end up affecting individuals that are not direct beneficiaries (whether this effect is intended or not). Hence, the design of an evaluation should understand the mechanisms in which these effects are possible not only to include controls that are “pure” (not affected indirectly by the program) in the survey design, but also to measure the transmission effect also relevant for policymaking.
The American Evaluation Association is celebrating Latina/o Responsive Evaluation Discourse TIG Week. The contributions all this week to aea365 come from LA RED Topical Interest Group members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.