RoE TIG Week: Evaluation Capacity Building Interventions & Sustainable Evaluation Practice: Research & Practice by Jay Wade

Hello, AEA folks!  My name is Jay Wade, and I am the Evaluation Director of Planning, Implementation and Evaluation Org (PIE Org), a non-profit consultancy in Chicago, Illinois that specializes in evaluation capacity building (ECB).  I have worked with a colleague, Dr. Leanne Kallemeyn of Loyola University Chicago, to study whether and how ECB interventions can lead to improved and sustained evaluation practice.  In our study, we interviewed 15 organizational leaders in North America whose organizations underwent an ECB intervention.  Here are our key findings:

• ECB interventions helped support sustainable practice.  All organizations continued to conduct and use evaluation after ECB.
• We found a common developmental pattern across organizations.  First, leadership was supportive of evaluation and advocated for building the organization’s evaluation capacity.  As a result, leadership dedicated resources to evaluation, such as hiring a consultant and investing in proper technology and personnel training.  With these resources in place, organizations were able to conduct evaluations, use evaluations, and develop evaluative thinking, which was bolstered by the personal factor of the ECB practitioner.  Once evaluation findings were used, staff were able to understand the benefits of evaluation to the organization and began to buy-in and value it. This buy-in then reinforced support for evaluation among leaders, which perpetuated this cycle.
• This cycle was iterative.  The ECB intervention and the role of evaluation practitioners with the organizations changed after multiple iterations of the cycle.

Lessons Learned:

So what does this mean?

• ECB practitioners can promote sustainability by:
  • Supporting leadership, particularly at the Board level.  Engage leaders and cultivate a champion to start.
  • Pointing out programmatic benefits of evaluation in real time so that staff can begin to value the process.
  • Aligning the work to the mission and values of the organization explicitly.  Use the organization’s own language.  The more staff “got along” with and saw the evaluator as on their team, the better the buy-in.
  • Shifting an ECB intervention overtime from intensive supports (e.g., professional development, logic model development, tool development, analyses processes) to technical assistance and coaching, only when needed.
• Researchers can build on this work by studying the following:
  • What does sustainability of ECB look like in a more diverse sample of organizations, including large organizations and organizations around the globe? 
  • The cycle above is iterative.  How do these organizations continue to develop their capacity over time?  What happens after years of iterations?  A decade? 
  • How do ECB practitioners develop evaluation champions and how does that work relate to sustainability?

We hope this gives practitioners some helpful validation and information; and we look forward to what the field learns in the future!

---

The American Evaluation Association is celebrating Research on Evaluation (RoE) TIG Week with our colleagues in the RoE Topical Interest Group. The contributions all this week to aea365 come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.