Hi, AEA365 family! I’m Sara Vaca (frequent Saturday contributor to this blog), independent consultant specializing in evaluation of international development programmes and policies, and I’m platonically in love with Participation. Here is my practical relationship with it so far, though.
Hot Tip: I understand participation as including many different actors, preferably those who have less voice, not as informants, but as part of the group that make decisions about the design and process, rather than just commissioners and evaluators. As a constructivist trying to bring an Equity and Gender focus to my evaluations, I understand participation enriches (or even legitimates) the process and results. So, since my first evaluations, I always pitch the idea of including beneficiaries (and other actors) into the evaluation team.
And every time I am about to face a new, during the inception phase my hopes go high on trying to make it really participatory. But after many unsuccessful attempts, my experience has kind of killed my hopes and I had that wise, yet sad realization: “It is most probably not going to work, as usual…”.
Lessons Learned: These are the main obstacles I have found in my practice that make the commissioner organization and the evaluation managers (and sometimes even myself) decline my suggestions to foster participation:
- Organizations do not understand participation as bottom-up: even when the Terms of Reference talk about high levels of participation, they often have been elaborated “in-house” (= without participation)
- Normally, participation makes the evaluation more expensive (unaffordable?): it entails logistics costs to make actors meet in centered places, to participate in meetings where things related to the evaluation are decided and defined, and longer times (more expensive again, in terms of evaluator’s fees and costs).
- Beneficiaries of international development programmes usually do not know what evaluation is: in contexts of low-income, low-literate, urban or rural communities who have problems to survive every day, involving them in a technical discussion about what to ask and how to answer evaluation questions is forced, hard and almost irrelevant to them.
- Ultimately, given the aid-vicious-circle and hardship they face, beneficiaries are busy and often engage into the process solely with the hope of getting extra aid, that is, the more they engage, give their time, the more they expect that will revert into new projects supporting them or their area.
Rad Resource: Forming Evaluation Reference Groups where you invite authorities, communities representatives and other organizations to comment on the ToRs, evaluation design and sampling, or at least to participate in a Findings Validation Workshop, are so far the two realistic mechanisms that in my view grant some participation minimums.
Cool Trick: However, I often use a Participation Scanner to depict the participation levels of the evaluation. And yes, until the majority of squares (or at least the ones at the right) is dark, no matter how many groups the evaluation consulted as informants, it is still not participatory.
Does the same happen in other contexts or types of evaluation? 🙂
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
4 thoughts on “Why (Real) Participation Is A Still a Pipe-dream (So Far ) In My Evaluation Practice by Sara Vaca”
I commend you for your honesty. This is too often the situation we face as independent evaluators. Evaluation reference groups work well although they are far from an ideal level of participation in many instances. As evaluators, we need to work harder to influence evaluation terms of reference so that they include an appropriate level stakeholder participation and the resources needed to achieve it. We also need to do more up-front work to explain to our clients what participation means and to convince them of its value.
Totally agree, Mark.
Thanks for your comment.
Great post and I love the visual you’ve created. I have a question though (because I struggle with this as well)…if you hold as a value that evaluation should be participatory, how do you justify doing work that is not participatory? I feel that our work is often not truly representative of participants and therefore not fully reliable. Yes, we do the best we can with the resources we are given, but I wonder sometimes if we do more harm than good by delivering results that are not grounded in our values.
Wow, Libby, thanks for such a deep thought. Apologies for taking long answering.
There are many ethic conflicts we (I) have to deal with every day, one of the biggest for me being the fact that my daily fees the organization pays for me are obscenely high compared to the lifestyle of the population they (and me) are trying to help… So yeah, I truly believe in participation but if it is not possible, I trust my contribution gives value (at least to the organization), and I know it does (because they tell me). Now about the validity of the whole development cooperation system… I totally agree on your doubts.