Greetings! This is Masahiro Igarashi, Director of the FAO Office of Evaluation and Vice-Chair of UNEG. and Maria Alice Christofoletti (Economist/Evaluation Consultant), sharing about the cost-effectiveness of having technical experts (e.g., agronomists, biologists) as part of the evaluation team. Despite being a frequent choice made by some organizations, the added-value of technical specialists in evaluations has not been the subject of much critical debate. In practice, their recruitment is often done without discussion of the potential issues involved. Evaluation literature – such as studies by Mary Kennedy and Boris Volkov – mainly focus on advantages and disadvantages of using internal or external evaluators and therefore, do not touch upon this subject.
The rationale for employing experts is, since some interventions are premised on specialized knowledge, they would require expert capacity to be properly understood and adequately assessed. Others would argue experts bring more credibility against the projects’ counterparts, who are technical experts themselves. At the same time, using experts in evaluation could lead to an assessment based on technical opinion rather than on actual results achieved, causing a con?ict of opinion between experts from the evaluation and project teams.
We take advantage of a recent change made in the FAO Of?ce of Evaluation to respond to this question. Aiming at limiting the risk of evaluations becoming opinion-based rather than evidence-based, the Office revamped its evaluation approach in 2014 to a more results-based one.
Based on an internal survey, we found that participation of technical experts in the evaluation team has been most useful in data collection and analysis stages, and less so in evaluation design or report writing. According to evaluation managers, the presence of experts allowed probing on technical aspects that evaluators in general were not able to see, and also provided credibility to the team when dealing with counterpart. In a few cases, the subject matter was so technical that experts were indispensable in understanding and interpreting data.
The fact that technical experts were less used in designing evaluations or writing reports suggests that evaluation managers need to carefully calibrate the ‘technical expert’ role within the team. Depending on the intervention, technical experts could play a more useful role as advisors and validators at the design and report writing stages.
Survey results suggested that ?ndings from the evaluations could have been drawn by professional evaluators through various means – e.g., extensive collection of evidences and effective consultation with stakeholders. The use of technical experts would facilitate the data collection and analysis process and could add depth and breadth of ?ndings with their professional knowledge and perspectives.
We also found that hiring technical experts as part of an evaluation team sometimes increased costs but was generally proven valuable. Also, value-for-money was achieved by managers looking for local technical specialists rather than international experts. Therefore, findings suggest that using international experts should be limited to cases where high-level expertise is indispensable, such as for projects with high-level technical contents for which quali?ed local specialists are unavailable.
While survey results provided valuable findings to steer the Of?ce approach when hiring external technical experts, we hope this article can interest a broad group of evaluators and similar organizations that face similar dilemmas.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.