Hi, I’m Sara Vaca, independent consultant, helping Sheila curate this blog and occasional Saturday contributor. I haven’t been an evaluator for a long time (about 5 years now), but I have facilitated or been part of 16 evaluations, so I start getting over the initial awe of the exercise, and I am starting to be able to take care of other dimensions rather than just “surviving” (that is: understanding the assignment, agreeing on the design, leading the data collection process, simultaneously doing the data analysis, validating the findings, debriefing the preliminary results, finalizing digesting all these loads of information for finally packaging it nice and easy in the report).
I want to think that I incorporate (at least I try to) elements of Patton’s Utilisation-Focused Evaluation during the process, but until recently, my role as evaluator ended with the acceptance of the report (which is usually exhausting and challenging enough), taking no concrete actions once I had delivered it, partially because: a) it was not specified in the Terms of Reference (or included in the days of contract), or b) I usually didn’t have the energy or clarity to go beyond after the evaluation.
However, I’ve understood since the beginning of my practice that engaging in evaluation use is an ethical responsibility of the evaluator so I’ve just recently started doing some shy attempts to engage myself in it. Here are some ideas I just began implementing:
Cool Tick: Include a section in the report called “Use of the evaluation” or “Use of this report” in the document, so you (and them) start thinking of the “So what?” once the evaluation exercise is finished.
Hot Tip: Another thing I did differently was to elaborate the Recommendations section, but not in a prescriptive manner. Usually I would analyse all the evaluation ideas for improvement, and I would prioritize them according to their relevance, feasibility and impact. This time, I pointed out the priority areas I would focus on, and a list of ideas to improve each area, without clearly outlining what to do. Then I invited the organization to discuss and take that decision internally, and maybe forming internal teams to address each of the recommendations to gain more ownership.
Although, in occasions, clients have reached out months/years after the evaluation for additional support, this time I proactively offered my out-of-the-contract commitment to support, in case they think I could be of help later down the road.
Rad Resource: Doing proactive follow-up. I’ve read about this before, but haven’t yet done it systematically yet. So, I will set a reminder 3-6 months after the evaluation and check on how they are doing.
Hot Tip: I just published a post understanding Use and Misuse of Evaluation (based on this article by Marvin C. Alkin and Jean A. King), that helped me realize some dimensions of use.
As you see, I’m quite a newbie introducing mechanisms and practical things to foster use. Any ideas are welcome! Thanks!
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Sara. Thank you for writing this great blog post. Your suggestions on how to incorporate use within evaluation reports were simple, yet extremely effective. After all, it’s usually the simplest ideas that have the biggest impact on practicality. I appreciate how you mention suggesting improvements to a program in a non-prescriptive way. If I was one of the receivers of the report, I would likely use those suggestions to start talks within the organization and with other stakeholders – much easier than trying to come up with my own meeting agendas. I can see how these small changes in evaluation reports can have big impacts on a program’s future. They also shows the evaluator’s competence and overall usefulness. Important bonuses!
I also had an opportunity to read your other blog post, “Visual Display of Use and Misuse”. You actually helped me understand the differences between misuse, non-misuse, and misevaluation quite clearly. As someone who is just learning the terminology and the theory behind good evaluations (as part of my Master’s program), having visuals is key! Now I understand why it’s so important to follow up a few months after an evaluation is complete. An evaluator would not want to spend all that time and energy into producing a quality evaluation for the users to later misuse the report and its contents.
Again, thank you for your great insight and for sharing your knowledge.
Thank you Sara for reminding us that the responsibilities of the evaluator extend beyond the submission of the evaluation into the implementation stage. As you stated, evaluators are usually exhausted by the time they complete the evaluation process and often have a contract that finishes once the report is submitted and therefore they consider their job done.
Recently I watched a video, “Utilization-Focused Evaluation for Equity-Focused and Gender-Responsive Evaluations”, (https://www.youtube.com/watch?v=jQP1FGhxloY) in which the speaker, Michael Patton, who you mentioned in your blog, argued that the evaluator must not view his job as finished with the submission of the evaluation report but that it is incumbent on him to ensure that the findings are not misused or misinterpreted during their implementation. This is in line with your argument about the ethical responsibilities that the evaluators have to the stakeholders in terms of ensuring that the evaluation recommendations are acted upon, and you suggested some easy suggestions on how they can ensure that the recommendations they put forth are implemented.
Including a “use of the evaluation” section in the document, pointing out priority areas for the organization to focus on, and doing a proactive follow-up are not overly time consuming and the first two can be done during the written stage of the evaluation process. These three simple steps offer support to organizations where none may have existed before and decrease the chance that the evaluation report is unused or misused. An additional step that an evaluator should take to help facilitate the implementation of the recommendations put forth in the evaluation report, would be to prepare and work with the stakeholder to get them ready to accept the findings of the report. This also follows the Process Use approach, which advocates for the participation of the program’s stakeholders throughout the evaluation process. Is this something that you also agree with?
As a novice in program evaluation, I appreciate any suggestions and insights that the “experts” in the field can offer. I am also wondering if you have seen any improvements in the way your recommendations are being implemented and how receptive the organizations are to your follow-up inquiries since you have modified your practice.
Kind regards,
Erika Watanabe
Hi Sara,
I thoroughly enjoyed your post. I am new to the concept of program evaluation, although I have been witness to many consultation projects that sat dormant and were never utilized or implemented. One of my past employers had a total of six “strategic plans” that were sitting in the closet – all saying similar things about the non-profit organization and nothing was ever done with them. I think that the follow-up approach you are about to implement would be really interesting. I also love the idea of sitting down and working through prioritizations instead of outlining them yourself. What a great way to build interaction and an implementation strategy with stakeholders and users!
As you mentioned, you felt you were “surviving” the first five years of evaluation. Honestly, I feel like I’m doing a bit of that right now in my evaluation use course (part of my Master’s program). I’m intrigued by thought-leadership, such as Patton and Alkin, but I’m wondering if there are any resources you have found particularly useful in your transition from “surviving to thriving”?
Thank you in advance your time and assistance.
Yours,
Erica
Sara this post resonated with me and gave me a great way to illustrate to my clients how we can work together to successfully evaluate. I always appreciate your visuals!
Glad to hear that, Jenny!
Many thanks 🙂