My name is Pat Seppanen. I’ve worked as an evaluator now for more than 20 years. A good part of my practice has centered on evaluating complex change—initiatives designed to address human needs and community problems that do not fit into established program and policy categories. If concepts like non-linearity, emergence, adaption, uncertainty, and coevolution describe the work of your clients, I bet you have been having a tough time applying traditional evaluation designs. I’d like to offer a few tips and resources regarding evaluation of complex systems, hoping that you can use them in your work.
Hot Tips:
- Evaluation and consultation are inseparable in complex change initiatives and need to be integrated. In my experience the work is 50% evaluation and 50% consultation to facilitate successive iterations of collaborative problem solving and mutual learning. Since I am strongest on the evaluation side, I have joined up with someone who is an experienced facilitator and coach to do work.
- A situational analysis is a vital step to getting the work going. I usually build a situational analysis in as an activity in my work plan. The information you assemble will help you see patterns that will inform your design. For example, for one citywide initiative we are working on, the evaluation design is organized by the major buckets of work and different types of data are generated based on the information needs of stakeholder groups operating at different levels of the system. In another initiative, we have organized data collection in terms of the different levels of the system: national, community, center, program, and individuals. In doing a situational analysis, I use versions of Human Systems Dynamic tools developed by Glenda Eoyang (see the resource below).
- A complex initiative may include components that are simple, complicated, and complex (see Chapter 4 of Patton’s book that I’ve listed as a resource below for a great discussion of the characteristics of these components)—some components may not benefit from developmental evaluation while others will. If the focus is on improving and stabilizing the component, then a formative evaluation design is needed. But if the focus is on “learning by doing” then I’d propose a developmental evaluation design
Rad Resources:
I look to Michael Quinn Patton’s book, Developmental evaluation: Applying complexity concepts to enhance innovation, as my primary resource.
A much shorter monograph by Hallie Preskill and Tanya Beer titled Evaluating social Innovation very concisely offers lessons about doing evaluation to support adaptation.
If you are looking to learn more about complex adaptive change I recommend looking into Human Systems Dynamics.
The American Evaluation Association is celebrating Minnesota Evaluation Association (MN EA) Affiliate Week with our colleagues in the MNEA AEA Affiliate. The contributions all this week to aea365 come from our MNEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.