I’m Kylie Hutchinson (a.k.a. @EvaluationMaven), independent evaluation consultant and trainer with Community Solutions Planning & Evaluation and author of Survive and Thrive: Three Steps to Enhancing Your Program Sustainability.
The word I chose to memorialize this week is a small but important one. It’s the letter “a”, as in “She asked for a final evaluation report”.
As evaluators, many of us are accustomed to providing a single (and lengthy) final report at the end of the evaluation. However, change is in the air, and many of us would also like to see the demise of the final report because it often goes unread by busy decision makers and sits on a shelf collecting dust. But a two-page briefing note doesn’t work in all situations either. Clearly, one type of report does not fit all, which is where the concept of layering comes in.
Hot Tip: Layering is a term I coined in 2008 to describe the simultaneous use of diverse report formats to communicate your evaluation results. The purpose of layering is to give different stakeholders the option to go as shallow or as deep as they choose into your evaluation findings. It works like this. Imagine a lengthy report as the meat of a burger; it can be very heavy and take a long time to digest. Final reports are often very dense documents, and not all stakeholders have the time nor appetite to eat them. Sometimes, all they want is lettuce with a bit of tomato (e.g. a newsletter), or a slice of cheese (e.g. a podcast). Or they may be rushed and can only take a quick nibble of the bun (e.g. an infographic). Some of these users will be intrigued enough to eat the whole burger, appendices and all, while others might be satisfied with just a few bites. Layering works because each communication product contains the same key messages and is linked to a more detailed option, enticing the reader to learn more if they choose. By employing diverse communication strategies for these varying appetites, you give intended users the choice of how deeply they wish to delve into the results.
Rad Resource: A Short Primer on Effective Evaluation Reporting. In this upcoming book, I talk more about the concept of layering and present different ideas for communicating results beyond the traditional lengthy report.
The American Evaluation Association is celebrating Memorial Week in Evaluation. The contributions this week are remembrances of evaluation concepts, terms, or approaches. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Kylie,
I am a teacher who is currently enrolled in the Queens’ University Professional Masters of Education Program. Currently I am taking a course on program evaluation design, and one of our tasks was to find an article of interest on the AEA365 blog and contact the author. After a lot of searching, I stumbled across this article. I found it especially interesting because in my own program evaluation design that I am creating for this course, the reporting of data was something that I hadn’t really considered at length. I was too preoccupied with becoming familiar with technology, learning how to use a logic model, and learning theories of evaluation. However, when it came time to explain how I planned to report data, suddenly I drew a blank. I had overlooked something that was essential if I expected to promote evaluation use.
In the past few weeks of this course, my peers and I have been talking a lot about collaborative evaluation and how it “has the side benefit of helping program people reflect on their practice, think critically, and ask questions about why the program operates the way it does” (Weiss 1998). Additionally, a collaborative approach “tends to aspire toward more equitable power relationships between evaluators and program practitioners” (Shulha and Cousins 1997). This is where your article really got me thinking: if program practitioners are only presented with a lengthy, intimidating report, how is that promoting an equitable power relationship? Further, how detrimental could this traditional approach of presenting one lengthy report be to evaluation use?
Your solution of layering seems like a great strategy to promote equitable power relationships, because it allows everyone to have access to data in a way that best suits them. As a teacher, I am constantly thinking about how my students learn differently and are at different levels, and this layering strategy really parallels how I typically structure learning in my classroom. I think that is one of the reasons I was so drawn to your article. Additionally, since everyone can access the data in a way that best suits them, evaluation use is promoted because it is more likely that more people are familiar with the data compared to if everyone had to read a lengthy report, and therefore they will be more likely to use that data productively.
My question for you is, does the strategy of layering take a lot more time? I know that when I am creating opportunities for different styles of learning in my classroom, my prep time increases dramatically. The increase in time is definitely worth it though because the benefits are enormous. Would evaluators take on a similar mindset? Finally, would this effect cost because of the increase in time?
Thanks for posting such a great article. It has definitely got me thinking more about data reporting in evaluation!
-Caitlin
Here are the articles that I referenced above:
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19, 21-33
Caitlin,
First of all, my apologies for only responding now. I just stumbled across your comment yesterday.
Your question, does layering take more time? My short answer is “it depends.” More than half of my clients these days have opted not o receive a long report at all. Ideally, the hours you would have spent writing a long report would go into several shorter alternatives. But it really depends on the situation. I can prepare a two-pager in a day or so (assuming my analyses are all done), but largely because I now have a template that I modify for each client.
Many evaluators consider a slide presentation a standard part of their reporting so that wouldn’t figure into the extra hours. Also, if you prepare a communication plan in advance, you can budget for the time and resources required. If you engage with your stakeholders appropriately, you can also determine what types of communication products are best to use for layering.
For more info, check out http://communitysolutions.ca/web/evaluation-reporting-guide/.
Good luck! I was in Kingston in June and it’s a beautiful city.
Kylie
Kylie, thank you for coining the term layering which I have long identified as a best practice.
Cheers,
Chad
Thanks Chad. The hamburger analogy doesn’t work for everyone, but the layering term seems to.