My name is Susan Kistler. I serve as the American Evaluation Association’s Executive Director and I provide each Saturday’s post for aea365.
Rad Resource: This week, I am reading Atul Gawande’s The Checklist Manifesto: How to Get Things Right. It is a paean to monitoring if not evaluation. Gawande weaves a tale of how checklists improve processes, from building skyscrapers to preventing infections in hospitals. He tells a compelling story, of lives saved and tragedy averted, and he backs up his reflections with data, data, data.
Hot Tip: Want to win a copy of The Checklist Manifesto? Leave a note in the comments indicating what reading has recently influenced your work and why.
Lesson Learned: Gawande draws on the work of Zimmerman and Glouberman’s proposition that there are three types of problems in the world: Simple, complicated, and complex. Simple problems basically can be learned by rote by a single person. Complicated ones often take multiple people and must be broken down into multiple related, coordinated, simple tasks to be performed. Solutions to complex problems can’t fully be learned – the context and factors are ever-changing and variable. His example of raising a child is a telling one of complexity – the lessons from your first do not fully prepare you for the second, but they do make your more prepared than you were when Child One arrived.
Lesson Learned: To my surprise, Gawande makes a very compelling argument that checklists have a place even (and perhaps especially) among the most complex of problems. They provide a means for harnessing variability, lending structure and guidance amid what may be chaos, and controlling risk if not eliminating it. He explains the ways in which checklists can be empowering – providing line workers with the information needed to make decisions and the leeway to act rather than be stymied by a bureaucracy awaiting permissions from superiors. The most ‘a-ha’ moment I had while reading was Gawande’s indication of how checklists might detail not specifically what to do but rather, when the unexpected strikes, with whom to collaborate, connect, and communicate in order to leverage the collective knowledge of a group to respond and resolve.
I’m off to revise AEA’s emergency response plan. And I highly recommend The Checklist Manfesto. It has changed the way I look at complexity. While admittedly leveraging for better or worse my love for logistics, I believe it has something to say to anyone leveraging data in the service of improving complex systems.
The above opinions are my own and not necessarily that of AEA. This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.
Enter into the comments boxes below a note indicating what you have read recently that has influenced your work and why. From among all those making comments within these guidelines, one name will be randomly drawn to win a copy of The Checklist Manifesto on Friday, September 30, 2011. To be considered, your entry must be posted on or before midnight ET on Thursday, September 29.
Susan, I was in the middle of reading “The Checklist Manifesto” when you posted about it and wanted to finish reading the book before commenting. In addition to what you wrote about its usefulness to evaluators, I had some other takeaways from the book. In the sense of formative evaluation, we can be very helpful to our clients if we can help them identify the key steps and questions in whatever complex, repeatable processes they go through to achieve their intended results. It’s asking ourselves what should go on their checklists. What a valuable service this would provide! Also,program evaluation is often a complex and repeatable process that deserves its own checklist. Much like Gawande’s example of surgery teams, evaluation often involves a range of stakeholders in a temporary team. As Gawande describes in his book, a key value of the checklist is in getting everyone to act like a team by having them agree to focus on a common set of procedures and questions. The checklist can give evaluation stakeholders that focus and also the confidence that they can manage a difficult situation together, such as we do when we help our clients achieve their goals in very challenging social, economic, and political environments.
Drumroll please….all the names are in a [virtual] hat. She reaches in, rummages around, and pulls out – Jonathan!
Thanks to everyone who contributed, and Jonathan the Checklist Manifesto is on its way to you from Amazon.
One book that I have recently read that has influenced the way that I work is The Immortal Life of Henrietta Lacks by Rebecca Skloot. It is a fascinating read about the origins and the very human side of the infamous HELA cells used in research around the world.
This book has reminded me of the importance of considering humanity and the very real lives and communities that may be affected by the work that we do. It is very easy to get caught up in the methodology, the logistics of implementation and the collection and analyses of data involved in complex evaluation studies/projects. This books reminds us to also strongly consider ethics, cultural competence, how we collect data, and how we communicate our findings in a meaningful way.
I have been slowly reading and digesting the profound insights found in Dietrich Bonhoeffer’s book Life Together. For those not familiar with Bonhoeffer, he was one of (if not the only) German theologian to decry Hitler during the early years of his ascendancy to power in Germany. Bonhoeffer returned to Germany from the U.S.because he felt it was his duty to speak up against the Third Reich. He and other family members were executed just days before the liberation for their participation in an attempt on Hitler’s life. Life Together has great applicability to the practice of living and working in harmony with others.
I am reading More Than Good Intentions: How A New Economics is Helping to Solve Global Poverty.
Last year I read Marilee Goldberg’s “Change your Questions, Change your Life”. It opened up a new side of thinking and facilitation for me as a previous lab scientist and has changed my evaluation practice emensely (hopefully for the better!).
I have also just finished reading “Made to Stick” by the Heath brothers which provided some great guidlines/tips in communicating/translating my evaluation work (see also other post above).
Hi – thanks for this positive review of the Checklist Manifesto. I’m afraid I’d assumed it was stuff I already knew, but it sounds pretty rich!
Personally, I’m reading the following books now and finding them all very useful:
Getting to Maybe: How the World Is Changed, by Francis Westley et al (good inspirational text for those days when it feels like one is alone facing a brick wall)
Systems Concepts in Action: A Practitioner’s Toolkit, by Bob Williams and Richard Hummelbrunner (this was recommended to me by another AEA member as a classic in the field. I didn’t know it, so am happy to see much of what I already do in my research as a matter of course given such a useful framework and set of vocabulary terms)
Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Michael Patton (much is familiar here, but I really appreciate his bringing evaluation into conversation with complexity theory)
I recently read Checklist Manifesto and I have to agree with Susan that it has many ideas applicable to evaluation. Another book that isn’t directly abut evaluation but has many translatable ideas is “Made to Stick: Why Some Ideas Survive and Others Die” by Chip and Dan Heath. The authors describe 6 characteristics that lead to ideas “sticking” in people’s minds: 1. Simplicity; 2. Unexpectedness
3. Concreteness; 4. Credibility; 5. Emotions; and
6. Stories. When presenting information or telling evaluation stories I find that these characteristics are good to keep in mind.
The most influential book for me in the last few years has been slide:ology by Nancy Duarte. Not specifically about evaluation, but rather about presenting information (not just data) in a way that facilitates real change – it’s changed the way i interact with my information and how i share it with my partners. I HIGHLY recommend this book – it’s amazing.
Like others who have posted so far, I am reading a book that is currently influencing my work, A Guide to Manage Knowledge:Cultivating Communities of Practice by Wegner, McDermontt and Synder. My work is looking at collaboration within a national network (particularly on part of the coordinating center) for a problem that is not only complex, but meets Rittel and Webber’s definition of “wicked”. This book has helped me conceptualize not only collaboration/partnership, but also value-creation and knowledge management as part of a constructs for measuring collaboration from a process (not structural) angle. We are also using it to develop a new program to offer in the network.
“Evaluation Methodology Basics” by E. Jane Davidson encourages evaluators to work with clients to identify a priori criteria for concluding that a particular finding is “good”. For example, the book suggests asking client to indicate what proportion of program participants, responding to a survey, would have to agree that a program was effective in order to conclude that the program was in fact effective. Clients are not always in a position to identify these criteria, but they usually appreciate the opportunity to think about them.
I’ve recently read, EPIC Change: How to Lead Change in the Global Age, by Timothy R. Clark. I found Clark’s model nails the inter-relatedness of what leaders need to be aware of, the energy level and attention focus needed to lead change, and the cycle of change. EPIC is an acronym for Evaluate, Prepare, Implement, Consolidate. I like how evaluation is put at the beginning of the process, and is not an afterthought. Clark states, “Leaders who aren’t engaged in change should be engaged in evaluation. In order to maintain competitiveness and fulfill the institutional mission of an organization, every leader has a responsibility to continuously evaluate three things: competitive reality, internal performance, and alternatives for change.” I use Clark’s in teaching evaluation to senior government leaders as another reason why evaluation is a mandatory management responsibility.
I’m almost through reading The Power of Positive Deviance: How Innovators Solve the World’s Toughest Problems by Richard Pascale, Jerry Sternin, and Monique Sternin. It’s about an unconventional approach to solving complex problems using the lessons learned by the “positive deviants” who live/work under the same conditions as others in their community/workplace yet find a way to succeed against all odds. The approach has been successfully implemented in over 40 countries to bring about positive change. I’m drawn to the approach because it relies heavily on qualitative methods to uncover the deviants. And it gives me hope in a world of limited resources.
One of the most insightful books I have read is The Careless Society: Community and its Counterfeits by John McKnight. Written in 1995, it’s a frank and stinging view of how communities are taken over by service providers for their own advantage and ends. According to McKnight, “The enemy is not poverty, sickness, and disease. The enemy is a set of interests that need dependency, masked by service.” In many ways, this is as true today as ever. The Careless Society is a powerful indictment of how successful outcomes are identified, determined, and distorted, by whom, and why. When I first read the book I was working in communities as a prevention practitioner and program manager. McKnight certainly gave me pause in thinking about the work we were doing and how we were measuring results. I highly recommend this book for anyone working in the field, as well as his new work, The Abundant Community: Awakening the Power of Families and Neighborhoods.
I’m reading Purposeful Program Theory by Sue Funnell & Patricia Rogers. Lots of practical ideas for evaluators and practitioners. I like the flexibility and recognition of the range of programs and contexts.
While it seemingly has nothing to do with evaluation, I will make a connection here. As a public school educator, one of the best books I’ve read in recent years is Lost at School: How Our Kids with Behavior Challenges are Falling Through the Cracks and How We Can Help Them. The author, Ross W. Greene, describes how a Collaborative Problem Solving approach with children has the adult(s) helping kids get THEIR concerns out on the table FIRST when discussing challenging behaviors, and then engaging in a collaborative effort WITH the kids to collectively create durable solutions to the problem. Hmmm. Seems to me this type of approach could serve an evaluator well, and Greene admits that the CPS approach is not just for children. To borrow his words… “Challenging behavior occurs when the demands of the environment exceed the individual’s capacity to respond adaptively.” How many evaluators have encountered challenging behaviors from adults when working with clients and stakeholders? What might happen if we allowed those folks to voice their concerns before we express ours, and then work in concert with them to develop solutions? Food for thought…