Hi everyone. I’m Isabelle Collins, evaluation principal at the New Zealand Social Policy Research and Evaluation Unit (Superu).
With increasing focus on a social investment approach, and the need for data to feed that, and moves towards outcome based contracting in the social sector, the role of evaluation (and evaluative thinking) is increasingly important.
Lesson Learned:
We found, while delivering training to service providers and contract managers that people were bemused by language (especially the cavalier way evaluation hijacks some words in common usage) and needed simple introductions to complex concepts that enabled them to have sensible conversations.
Rad Resource:
Based on the training, we have produced a free handbook on Making Sense of Evaluation an easy-to-read guide that will take you from the conceptual and theoretical to the firmly practical, accompanied by the cutest of penguins along the way.
Information is presented in four modules covering preparation, logic models, measuring impact, and commissioning and managing an evaluation. There are examples, diagrams, hot tips, case studies and lots of pointers to other, more detailed, references.
We find that the penguin case study helps people look beyond their (often difficult) daily experience and understand concepts before applying labels, so they can begin to adapt their practice to meet new requirements from a position of knowledge.
Rad Resource:
The handbook is just one of several resources for the community and voluntary sector on the Superu website, designed to help people build and use evidence for impact.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Dear Isabelle,
My name is Ryan Goldie. I am currently enrolled in a course called Program Inquiry and Evaluation as a student in the Professional Master of Education program at Queen’s University in Kingston, Ontario Canada. The purpose of this letter is to inform you that I found the resource you provided here very helpful in simplifying for me the process of program inquiry and evaluation, and that I appreciate this contribution which you have made to the field.
As I embarked on my studies of program evaluation I had much difficulty understanding the jargon and the process of PE despite spending a considerable amount of time reading the assigned text by Huey-Tsyh Chen called “Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness”. The handbook that you have produced does a fantastic job cutting through the jargon by listing the words next to other words that are used to identify the same thing. I also like how it provides definitions to the jargon in easy to read, colourful tables. It was also very helpful that the jargon is highlighted in boldface text as it is being used in context. The concept of a logic model is also clearly defined and outlined through the supremely simple exemplars and cases studies. The analogy of having a headache and how to relieve it as one way to demonstrate how to create a logic model was extremely simple yet effective in clarifying what a logic model is. In addition, the use of penguins as the target population for an example program made it feel like I (the reader) was being hand-held through the process of making a logic model, highlighting assumptions, and measuring success. However the real examples that are also given effectively bridge the gap between the overly simplified and realistically complicated. Just at a time when my course required me to make my own logic model for a program I intended to evaluate, reading this part of your document was encouraging to me as the textbook examples were a little difficult to decipher at that time.
I shared your handbook with my classmates recently and many of them expressed their appreciation for its effectiveness at simplifying the concept of program evaluation. Their testimonials are included below my closing salutation.
Thank you very much for help those of us new to the field of program inquiry and evaluation get on our feet and begin to walk. Being able to evaluate a program properly is truly key to helping make organizations run more efficiently and improving the experiences and/or lives of stakeholders. I feel your contribution through this handbook is an eloquent addition to the field that every evaluator must read.
With sincere appreciation,
Ryan Goldie
Testimonials:
Thanks for this great resource. I am working my way through it now, and I feel it is a great fast read that gives a quick and clear overview. I will no doubt return to sections of it as I build my own evaluation in the coming days. What I liked seeing is in the resource you provided is that often nested models are used to help solve this problem. For now, I will try to continue with a single logic model, but feel re-assured that a nested model is a possibility if the task gets to overwhelming. Tamara
Thank you for sharing this resource! I have also been having difficulty with the terminology and that has made it difficult to understand some concepts at times. The resource is easy to read and helps to clarify many terms. I like how it provides a simple explanation of each term and includes examples and diagrams. I will definitely refer back to this as I reflect on my theory of change and logic models as I continue to develop a better understanding of what everything means and how it all works together. Jenny
This truly is a great resource. I am hoping that through the reference of this handbook that I will be able to feel more confident moving forward with my evaluation. I really like how user friendly it is – where it is designed for the beginner, and doesn’t try to impress flowery language for the purpose of academia. It makes it far more accessible for more people who are thrown into the evaluation position or who are hoping to get more experience doing it effectively. In that respect it is a truly great resource! I also like that it is broken down into modules so that you can actually use it alongside your evaluation. It feels very supportive! Furthermore, not only is the handbook not overly “jargon heavy” but it includes some kind of jargon dictionary to help those reading it communicate better with other evaluators or policy makers. Lydia
Hello Isabelle,
I had the pleasure of stumbling across your blog posting on AES365- A Tip A Day, and I very glad that I have. First, let me introduce myself, my name is Jaclyn and I’m a Professional Masters Student at the University of Queens in Kingston Ontario, Canada, as well a second year primary- junior teacher. My Master’s focus is in assessment and evaluation, and with this I hope to one day be able to create evaluation and resources to help meet the needs of all of my students.
Something I’ve noticed in my first year of teaching in completing report cards, as well as my Master’s course readings is that people often get lost in the language that they miss the entirety of the theory or what the reader had intended them to understand. We have had several discussions during PD Sessions at my school about making the language on report cards ‘parent-friendly’ as they’re not going to understand the terms we use from the curriculum- they’re just teacher jargon. This is why I have found your blog posting to be so enlightening and valuable as it is easy to read and understand! (Not to mention the adorable penguins). You created something that ‘everyone’ can understand. I really appreciate how your handbook walks the reader through the steps, as well as clarifying the jargon of ‘what it means’ as well as ‘other terms used for it.’ Something else I admired about your handbook is how you simplified everything…including how to create the most basic logic model, which helped me as someone new to logic models understand how each step links to different things (such as output/ outcome), as sometimes I am like many other reads and have to reread something several times to get the gist of it.
I’ve just completed my first ‘Program Evaluation’ as part of my course, and as many of the concepts are new to me I wish I had found your resources sooner as it clarified so many things that I’ve been trying to wrap my head around this semester (including what process/ outcome program design are and how and when to use them). I absolutely love your resource and will be sharing it with my classmates and other educators.
Thank you for simplifying and for creating a guide that is for everyone!
– Jaclyn
Hello Isabelle,
Thank you for the important work you have done with Making Sense of Evaluation. My name is Muna AlZier, I’m currently a student at Queen’s University studying to earn my Professional Masters of Education. As a graduate student in the field of evaluation, even I am often overwhelmed at the emotional complexities of the field. I found your work extremely useful and relevant in addressing some crucial concerns in the field of evaluation such as the need for “simple introductions to complex concepts” in order to enable people to have conversations about evaluation and evaluator recommendations. One issue that I have seen arise in the literature and discussed in detail by Shulha and Cousins and other industry leaders is the emotional nature of evaluation as a field, which you referred to in your introduction.
As someone who aspires to join the field of evaluation, I think it is important to consider certain risks. From your research and my own, it seems that evaluators themselves can often become emotionally intertwined in problematic organizations and the participants can be reluctant, change their minds, or face consequences for being honest during the evaluation process. Pressure from various stakeholders can also have an emotional impact on evaluators and participants. However, your use of case studies and pointers seems particularly useful as evaluation is also a complex field that is not easy for the layman (or even the new grad student!) to understand.
Further, evaluation is such a difficult field and tool to describe to others, and people often misunderstand what evaluation really is. But, as you mentioned, you have written this work to serve as a resource for the community, and created a manual that uses real examples, or case studies, and presents them in a humorous way with penguin drawings! I think this work could really help those outside of the field of evaluation understand the purpose and use of evaluation in their professional lives.
Thank you for your contribution to the field, and for helping a grad student like me!
Muna AlZier
Try this: http://www.superu.govt.nz/resources/handbook
Hi Isabelle,
The link to your handbook doesnt seem to be working. Can you repost in comments?
Thanks!
Kari
Hi Kari
Thanks for your note. We’ll look into why the link isn’t working. In the meantime please find a link here.
http://www.superu.govt.nz/resources/handbook
Kind regards
Kay