AEA365 | A Tip-a-Day by and for Evaluators

TAG | evaluation advisory groups

AEA365 Curator note: Back in January, AEA365 readers asked to read about how evaluators deliver negative findings to clients and stakeholders. This week, we feature 5 articles with four evaluator perspectives on this topic. 

Hello AEA 365 readers! I’m Glenn Landers, the Director of Health Systems at the Georgia Health Policy Center (Andrew Young School of Policy Studies, Georgia State University). A large portion of our work is evaluation, and we’ve been fortunate enough to work in every state and many of the territories. No one likes being the bearer of bad news, but sometimes it can’t be helped.

Recently, I was engaged in a developmental evaluation of a collective impact initiative that was intended to last ten years with ample funding. Five months in, we realized the initiative was in trouble. One year in, the project was basically over. Several techniques helped incorporate the bad news into the process as learning.

Hot Tip:

Evaluation Advisory Groups! We always try to have an advisory group made up of those whose work is being evaluated and those who will use the products of the evaluation. This way, we can test what we are learning with a small group for feedback before sharing with a wider audience.

Hot Tip:

Feedback loops! We also set up several feedback loops with the funder, the facilitator, and the work’s steering committee. This way, we shared information in small packets and gained the benefit of group sense making so that everyone understood why things weren’t working as planned.

Hot Tip:

Evaluation as Learning! We were fortunate to have a project sponsor who was interested in learning from what was not working just as much as what was working. Knowing this upfront helped us to be more comfortable in being candid.

Lesson Learned:

There’s nothing that can be substituted for being present with the people who are doing the work. Relationships and trust develop over time. The more present you are with them, the more they will be able to be in a position to hear the results – whether good or bad.

What’s worked for you in delivering bad news?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

We are Sonia Worcel, Vice President of Strategy and Research, and Kim Leonard, Senior Evaluation Officer, at The Oregon Community Foundation (OCF). The research team at OCF conducts internal evaluations as well as oversees contractors for some external evaluations. We use Evaluation Advisory Groups (EAGs) for our internal evaluations, and currently have EAGs for three varied evaluation projects: a developmental evaluation, a policy/systems change evaluation, and a more traditional process and outcomes evaluation.

Hot Tip: Evaluation Advisory Groups have great potential to enrich the design and implementation of evaluations, and can support and enhance the use of evaluation results. In addition, the groups also provide external, impartial oversight to the work.

Our advisory groups serve as a sounding board providing valuable feedback on ideas and plans for evaluation design and methodology. For example, one EAG has provided valuable feedback about the design of a comparison group component of one evaluation. Another EAG weighed in on the development of several key evaluation data collection tools including a survey and photo voice component.

Hot Tip: Select members of an EAG in order to represent a depth and breadth of backgrounds and expertise, including content-area experts, evaluation methodology experts and practitioners. This will position the EAG to offer a variety of perspectives on the work and can complement and build upon the expertise of the evaluation team.

Lesson Learned: Our EAGs have connected us with resources and experts that in turn have allowed us to further refine the evaluations and contribute to the larger field. Some EAG members have spent additional time to share related work or to review tools or results in greater detail. One of the EAGs has come to feel like a professional community of practice, as many members are now working together on other related evaluation and research efforts – building shared measurement and design in one particular content area.

As each evaluation progresses, we anticipate that the EAGs will play a growing role in supporting evaluation dissemination and use. One EAG recently spent a longer mini-retreat meeting reviewing early findings in detail alongside the Research team, resulting in rich discussions about how best to share results with grantees and valuable feedback about what felt worthy of further analysis and broader dissemination.

Rad Resource: The Winter 2012 Issue of New Directions for Evaluation featured articles about Evaluation Advisory Groups. We found the advice included in those articles useful as we designed and continue to adapt our EAGs, and this advice is reflected in the tips and lessons learned above.

The American Evaluation Association is celebrating Oregon Community Foundation (OCF) week. The contributions all this week to aea365 come from OCF team members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Archives

To top