I’m Urmy Shukla, Capacity Building Manager at the CLEAR South Asia Regional Center, hosted by J-PAL South Asia at the Institute for Financial Management and Research. Since our 2011 start with CLEAR we’ve developed a wide-range of activities aimed at improving monitoring and evaluation (M&E) capacity throughout the region, including 90 trainings for partners such as the Indian civil services, state governments, NGOs, donor agencies, and academic institutions. Each training requires a significant amount of planning and preparation, including a needs assessment to assess skills and the partners’ role in evaluation, the development of customized content, and delivering the course itself. As such, we want to ensure that are trainings are meeting their objectives.
How do we know if our trainings are ‘working’?
As evaluators, we know that there are several steps to plan for, and later assess, effectiveness of our activities. Most importantly, we need to:
- define a theory of change and/or results framework for program activities, focusing on desired outcomes
- measure/assess the desired outcomes
For evaluation capacity development, these aren’t always easy to design and implement. But we’re taking several steps to assess the effectiveness of our trainings, including developing an organization-specific results framework and tracer surveys to track past training participants. We’re testing our approach as we’re going, and below are sharing some practical and strategic tips.
Hot Tips: For training tracer studies:
- Clearly define training objectives from the outset. These objectives should go beyond skills gained, but should also include what you hope the participants will do after the training, within what is reasonably feasible during that timeline.
- Develop a way to systematically organize your multiple objectives. This will make it easier for you to design future tracer surveys and needs assessments. We categorize our objectives by (a) partner type (those who either do evaluations, use evaluations for decision-making, fund evaluations, and/or commission evaluations) and (b) knowledge, attitude, or behavior (KAB). From this, we have developed a database of tracer survey questions, which can be easily filtered for each type of training.
- Get partner buy-in early. Getting people to participate in a tracer study a year or two after the training can be hard, so give advance notice at the training that a tracer study will occur. Then have some contact with trainees – through newsletters, announcements, listservs – after the training to keep contact info current and so they remain familiar with you.
Rad Resources:
While not M&E focused, this Helvetas Measuring Education’s Path to Prosperity: A practical Toolkit for VET Tracer Studies guide is a useful resource for tracer study planning.- Check out our own Interactive Course Manual: A Manual on Organizing and Running a Successful Training Course on Evaluation Methods.
The American Evaluation Association is celebrating Centers for Learning on Evaluation and Results (CLEAR) week. The contributions all this week to aea365 come from members of CLEAR. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.