Greetings! This is Masahiro Igarashi, Director of the FAO Office of Evaluation and Vice-Chair of UNEG. and Maria Alice Christofoletti (Economist/Evaluation Consultant), sharing about the cost-effectiveness of having technical experts (e.g., agronomists, biologists) as part of the evaluation team. Despite being a frequent choice made by some organizations, the added-value of technical specialists in evaluations has not been the subject of much critical debate. In practice, their recruitment is often done without discussion of the potential issues involved. Evaluation literature – such as studies by Mary Kennedy and Boris Volkov – mainly focus on advantages and disadvantages of using internal or external evaluators and therefore, do not touch upon this subject.
The rationale for employing experts is, since some interventions are premised on specialized knowledge, they would require expert capacity to be properly understood and adequately assessed. Others would argue experts bring more credibility against the projects’ counterparts, who are technical experts themselves. At the same time, using experts in evaluation could lead to an assessment based on technical opinion rather than on actual results achieved, causing a con?ict of opinion between experts from the evaluation and project teams.
We take advantage of a recent change made in the FAO Of?ce of Evaluation to respond to this question. Aiming at limiting the risk of evaluations becoming opinion-based rather than evidence-based, the Office revamped its evaluation approach in 2014 to a more results-based one.
Based on an internal survey, we found that participation of technical experts in the evaluation team has been most useful in data collection and analysis stages, and less so in evaluation design or report writing. According to evaluation managers, the presence of experts allowed probing on technical aspects that evaluators in general were not able to see, and also provided credibility to the team when dealing with counterpart. In a few cases, the subject matter was so technical that experts were indispensable in understanding and interpreting data.
The fact that technical experts were less used in designing evaluations or writing reports suggests that evaluation managers need to carefully calibrate the ‘technical expert’ role within the team. Depending on the intervention, technical experts could play a more useful role as advisors and validators at the design and report writing stages.
Survey results suggested that ?ndings from the evaluations could have been drawn by professional evaluators through various means – e.g., extensive collection of evidences and effective consultation with stakeholders. The use of technical experts would facilitate the data collection and analysis process and could add depth and breadth of ?ndings with their professional knowledge and perspectives.
We also found that hiring technical experts as part of an evaluation team sometimes increased costs but was generally proven valuable. Also, value-for-money was achieved by managers looking for local technical specialists rather than international experts. Therefore, findings suggest that using international experts should be limited to cases where high-level expertise is indispensable, such as for projects with high-level technical contents for which quali?ed local specialists are unavailable.
While survey results provided valuable findings to steer the Of?ce approach when hiring external technical experts, we hope this article can interest a broad group of evaluators and similar organizations that face similar dilemmas.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
3 thoughts on “Should we have experts in evaluation teams? by Masahiro Igarashi & Maria Alice Christofoletti”
Hello Masahiro and Maria,
I found your article very interesting, especially because I have recently written a short essay on the role of evaluators as a dilemma in program evaluation. Your topic connects to some thoughts I had. I appreciate that you said “their recruitment is often done without discussion of the potential issues involved” because I myself assumed that experts would be beneficial to have on an evaluation team without really thinking more in-depth about it.
As you mentioned, there are many obviously reasons why experts would be beneficial, such as experts understanding certain aspects of the program on a deeper level.
As you mentioned, I think it’s important to consider the negatives about adding experts when creating a team. Data collection and interpretation can get technical and it makes sense that having an experts help would be beneficial. According to your article, it may not be as helpful to have them actually determine the program’s success but used to add credibility and to help specifically with data. It then got me thinking, do you think it’s the evaluators role to decide who is on the evaluation team or do the stakeholders and program users should have input? The evaluation manager has the responsibility of deciding each member’s role which makes sense. However, you then mention that cost could be a factor and then, in that case, the person requesting the evaluation may have to have input on who joins the team.
Your article was thought provoking and interesting. Thanks for the read!
Thanks, Tanya for your thoughtful comments. I am glad that it provoked your thoughts.
On your question, the choice of the team members affects the independence/impartiality. So, in my view, it should depend on how the governance of the evaluation is set up. For instance, if the evaluation manager is in an independent and professional evaluation office, s/he could be entrusted by the stakeholders to develop the most suitable design and build the most suitable team to perform the task (within the budget provided). If not, an external team leader – presumably independent and competent – could be asked to do so. There is a possibility to involve the stakeholders and the programme users but we need to be careful for them not to have any conflict of interest. Often, they do have. So, I would limit their role in the design stage to providing inputs to the evaluation questions and the scope.
My two cents,
explain acronyms –what is UNEG????