My name is Susan Jenkins. I am a Social Science Analyst with the U.S. Administration on Aging and on the board of the Eastern Evaluation Research Society. I have found technical advisory groups (TAGs) invaluable for successful evaluation implementation. A technical advisory group should be made up of relevant subject matter experts who can provide technical advice regarding the methodology, implementation and dissemination of findings of your evaluation project.
Hot Tip: A TAG can keep your evaluation on track.
- Make sure to get input at key decision points in the evaluation and for important deliverables.
- Develop a schedule and process for gathering TAG input. While much of the feedback can be gathered by phone, webinar and e-mail, if possible, have at least one face-to-face meeting. I have found that such meetings set the tone for group interaction and encourage more participant commitment.
- Request both structured and unstructured feedback. While there will be specific documents you want reviewed or approaches that you want comments on, make sure to ask some open ended questions to get feedback in areas that you may not have realized could use improvement. I recently had to redesign an evaluation based on insights from a TAG. Specifically, the TAG pointed out that the time frame we envisioned was at odds with the grant period for the program. If we had stuck to focusing on how to make the existing methodology better most of the key informants would have moved on by the time we would have wanted to talk with them.
Lesson learned: TAGs can be helpful in a range of areas including:
- Ensuring that you understand the fundamentals of the program from people in the field, researchers, funders and policy makers. Make sure to include a range of people with practical knowledge and the time to devote to the TAG rather than just “big names”.
- Building buy-in for the evaluation. When asking for data I make sure to let respondents know that the tools and approach were developed (at least in part) by respected experts and/or practitioners in their field. I also use TAG feedback to shape stakeholders’ expectations and, when needed, to soften the blow of negative evaluation findings.
- Supporting dissemination of findings. A well chosen TAG will have contacts and real-world experience to guide the targeting of reports/products and their dissemination.
Rad Resources:
- The Robert Wood Johnson Foundation’s Evaluation Series report titled A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions.
- Program Evaluation Tip Sheet #33 from the Penn State Cooperative Extension provides tips for increasing TAG member interest in the feedback process.
- Papineau and Kiely’s article Participatory evaluation in a community organization: Fostering stakeholder empowerment and utilization published in Evaluation and Program Planning, is also a useful resource.
The American Evaluation Association is celebrating Washington Evaluators (WE) Affiliate Week with our colleagues in the WE AEA Affiliate. The contributions all this week to aea365 come from our WE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.