Hello! I’m Ann Marie Castleman with the University of Cincinnati Evaluation Services Center. In evaluation, we often talk about people becoming accidental evaluators. At one point in my evaluation career, I was an accidental evaluation commissioner. I didn’t know what evaluation commissioning was until I was tasked with managing my first external evaluation!
Evaluation commissioning is the process of putting forth a request for evaluation proposals, hiring an external evaluator, and managing the evaluation process. Evaluation commissioners or evaluation managers are important players in the world of evaluation. As the individuals who purchase evaluation services, commissioners play a role in shaping evaluation practice. Yet much of Research on Evaluation (RoE) examines evaluation theory and practice from the perspective of evaluators with relatively little from that of commissioners.
Lessons Learned
In my dissertation research, I set out to explore the capacity to commission evaluation. I examined individual, organizational, and systems level factors that influence the process and commissioner decision making. I surveyed commissioners working for international nongovernmental organizations (INGOs) and conducted four organizational case studies to dive into the details. Here’s what I learned:
Individual Level
At the individual level, there’s a relationship between the role of the commissioner and their prioritized evaluation use.
For instance, commissioners who were primarily program staff with commissioning responsibilities often prioritized instrumental use because they were interested in improving programs. Commissioners in managerial roles often prioritized conceptual use because they valued strategic learning across programmatic portfolios or geographical regions. Those in evaluation positions prioritized process use because they were building evaluation capacity internally – this was certainly the case when I was a commissioner.
Organizational Level
At an organizational level, the commissioning process was influenced by organizational evaluation policies that prescribed how evaluation should be commissioned.
Examples included specifying when to commission an evaluation in a program’s lifecycle and what evaluation criteria to include in the RFP. These were not always within the commissioner’s control to change.
This was true in my experience as well. The INGO I worked for had a schedule for conducting evaluations which fed into a learning agenda. While there was some flexibility, we tried to stick to the schedule so others could use the findings in the planned timeframe.
Systems Level
At a systems level, the evaluation market and donor requirements often influenced commissioner decision making.
Commissioners were challenged to find evaluators with the desired qualifications for a given evaluation (e.g., subject matter expertise, familiarity with the geographical region) who could design and produce a high-quality evaluation within a set budget by a specified deadline. Budgets and timelines were not always determined by commissioners, but rather by external donors or INGO head offices. These factors often reduced the size of the evaluation market available to a commissioner.
In my experience, finding local evaluators in the countries where evaluations took place was often a challenge perhaps due to limited budgets. Despite publicly posting RFPs on multiple platforms, we frequently received only a handful of proposals.
While this research surfaced many factors that influence the commissioning process, one of the most important appeared to be the commissioner-evaluator relationship. Relationships grounded in good communication and flexibility appeared to produce better evaluations—or at least ones considered high-quality and useful from the commissioner perspective.
Rad Resources
For any accidental evaluation commissioners reading this, check out the Manager’s Guide to Evaluation available in English and French. It breaks down the commissioning process into 9 steps and provides links to additional resources.
Regardless of whether you’re a commissioner or an evaluator, I hope these learnings provide some food for thought for your next evaluation contract!
The American Evaluation Association is hosting Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
