I’m Roger Miranda, an Independent Evaluator (here’s my website), and author of Eva the Evaluator, a storybook for both young and old explaining what evaluation is. This summer, I facilitated a workshop at the University of Manitoba’s Summer Institute on Program Evaluation. For it, I solicited ideas via Evaltalk, the American Evaluation Association’s public discussion list, about the “Things Evaluators Don’t Like to Hear.” I received over 50 suggestions from Evaltalk members. Based on those, plus a few of my own, I put together a “Top 10 List”. At no time was this meant to be a rigorous academic exercise (e.g. I decided what to include; on the ranking etc.) – thus the list is highly subjective, though given the “type” of suggestions I got, the themes they highlight are shared by many. Entries 1, 2 and 7 are things said to me while conducting evaluations- the rest were derived from the contributions.
Lessons Learned: Top 10 List of Things Evaluators Don’t Like to Hear
10. “Our program has been implemented worldwide, in six languages, with over 50,000 beneficiaries; we need the evaluation report in two weeks and have a $3,000 budget. By the way, we didn’t collect any baseline data. Can you do it?”
9. “An evaluation won’t tell us anything we don’t already know.”
8. “We never got round to preparing a Logical Framework or a Theory of Change.
We’ve been busy implementing our program from Day one!”
7. “Our organization doesn’t need an evaluation policy; we figure things out as we go along.”
6. “There’s no need to waste time by testing the survey instrument. Responses don’t have to be that precise.”
5. “Your work is just a sham — we know it’s going to be used to justify cutting our program.”
4. “We are not really interested in the evaluation. We just have to comply with funder rules and send them a report. As a matter of fact, they get so many that they don’t even read them.”
3. “I know our programme had tremendous positive impact and your evaluation report is not telling us that. Change your report. Don’t forget that I am the person writing the check.”
2. “Let’s hope that we don’t get shot or kidnapped during our field visit.”
1. “Evaluation? What the heck is that?”
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
“We just want you to put together data and reports that show that our program is effective.”
Reported to me by a colleague after I shared the Top 10 list with her.
Another Things Evaluators Don’t Like to Hear is Atlanta & Pennsylvania schools cheating scandal!
I work in the Division of Evaluations and Management Services at the Bureau of Land Management and I have another groan-inducing item to add to your list of things evaluators don’t like to hear. We are a decentralized agency and many of our evaluations, whether they be based on the quality of management or program effectiveness, are multi-site evaluations and require logistical support from the office we are about evaluate. However, we have had uneven success in coordinating these visits over time, no matter how many months of advance planning we undertake. This affects our ability to interview appropriately selected random samples of program or office staff as we have to make do with staff that is available at the time of the scheduled site visit.
The item that we would like to list of things that evaluators don’t like to hear is the following:
11. “Evaluation site visit? Ooops. Sorry. We forgot that half of the relevant staff (or managers) to be interviewed will be involved in off-site training for those two weeks.”
I hope you will accept this addition as I am sure this will resonate with a few other members of our evaluation community regardless of what sector they belong to.