Hi! I’m Emily Gates, an assistant professor of evaluation, and I’m Francisca Fils-Aime, graduate student, in the Measurement, Evaluation, Statistics, and Assessment department at Boston College.
Interested in how evaluation can contribute to system change? Us too! This is a tough question for which there are no easy answers. But, we have a few lessons learned from our research on systems-oriented evaluation. This research includes a literature review, cross case analyses, and, most recently, a case study of how a health system change initiative uses evaluation.
Lessons Learned:
1. There’s no such thing as “a” system.
Hold up, you might be wondering how we as evaluators can contribute to changing systems if systems don’t exist. There are two lessons packed into this one. First, systems are representations of reality, not reality itself. We individually and collectively construct systems to make sense of our complex social world. Second, there is not ONE system. Rather, we each have different viewpoints on any system of interest and erect different boundaries around what is and is not part of that system.
2. Drawing, mapping, and modeling systems can be super helpful.
Since we each view the same system differently, we need ways to surface, share, explain, and critique different system interpretations. For this, some of the best tools are good, ol’ magic markers and large paper! Check out formats for Systems Diagramming from the Open University. These representations of systems can help program staff and evaluators better understand the complex situations and systems they seek to change. They are also useful for pinpointing where and how evaluation can contribute.
3. Change is not neutral, especially system change.
We may eagerly rally behind the rhetoric of system change. But, this rhetoric disguises the fundamental pluralism of perspectives, values, and interests that comprise our democracy. And, more importantly, it disguises the systems of privilege and oppression that maintain inequities and injustices. System change requires us to ask: For whom? By whom? To what ends?
4. Facilitating learning and valuing are critical.
Sometimes, evaluation gets cast in the role of answering decision-makers’ questions which often are about assessing performance and making improvements to existing programs. Answers to these questions may fall short in addressing what really matters – ensuring that programs are designed and implemented to maximally influence change, address inequities, and foster sustainable solutions to complex social issues. This suggests expanding the scope of initial questions and criteria to facilitate ongoing learning. Further, it requires that evaluators and stakeholders engage in collective valuing about the value of programs and how they should be changed.
Rad Resource:
For more practical how-to guidance on using systems thinking and approaches, check out: Systems Concepts in Action: A Practitioner’s Toolkit by Bob Williams & Richard Hummelbrunner
The American Evaluation Association is celebrating Research on Evaluation (RoE) TIG Week with our colleagues in the RoE Topical Interest Group. The contributions all this week to aea365 come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.