Hello AEA community, my name is Dr. Tjip Walker, and I would like to share some thoughts about boundaries.
I spent a decade at the US Agency for International Development coordinating efforts to embed systems thinking into that agency’s operations, including the way it monitored and evaluated its programs. Since then, I have set up a consulting practice to support those invested in systems change and become more active in the Systems in Evaluation TIG (SETIG). In all these settings, one of the biggest challenges is setting boundaries.
For systems practitioners, boundaries separate the essential from the less-so. For a mechanical system, such as an automobile, establishing the boundary is straightforward. But when it comes to human-centric systems, determining the appropriate boundary is far from obvious and can often be contentious as it depends on individual judgements about what are the essential factors that are interacting to yield a particular outcome.
Take reading scores for example. Those outcomes emerge, at very least, from the interaction between students and teachers. So is our system boundary the classroom? Some educators point to the vital role parents play in encouraging and modeling reading. So do we expand our boundary to include the home? Others stress the role of pedagogy, teacher incentives, educational resources, or income inequality. So is the boundary properly set at the school level? District? State? Nation? The answer is not obvious, but the choice matters greatly for both decisions about priority interventions as well as what we choose to monitor and evaluate.
The same boundary question arises when we consider the intersection of systems thinking and evaluation practice. Since the founding of SETIG 20 years ago, there has been an ongoing debate about what bounds a systems approach to evaluation. There is more consensus about what is outside the boundary than within. Peppering papers and presentations with terms like “systems” or “systemic” does not necessarily make them so. What defines a systems approach is a way of thinking that seeks understanding from the way things are connected and interact rather than from investigating a discrete element.
Rad Resources to Introduce Systems Thinking
Appreciating the full dimensions—and power—of systems thinking can be hard. As a start:
- Check out this guide to introductory resources, which includes links to short videos and games, as well as documents.
- Another video—not short but full of insights for evaluators—is the presentation that Michael Q. Patton and Charmagne Campbell-Patton introducing systems thinking.
Systems Approaches to Evaluation
Within the community of systems thinkers, there are two schools of thought about how to approach evaluation. The older perspective, and the one that motivated the founders of a systems in evaluation TIG, holds that a systems approach to evaluation should be grounded in four concepts drawn from the broader field of systems thinking: interrelationships, perspectives, dynamics, and (our old friend) boundaries. In 2018, SETIG released Principles for Effective Use of Systems Thinking in Evaluation as a guide to this approach.
The second school of thought about systems and evaluation treats a system as the evaluand. This approach resonates with those who are committed to supporting systems change and are thus interested in evaluating if their efforts are having the desired effects. This approach does not deny the importance of perspectives, relationships and boundaries, but is more focused on understanding the dynamics of a system and how it is changing.
Rad Resource
Capturing and evaluating systems change can seem daunting. At USAID, we developed the 5Rs Framework as an accessible tool for describing the key features of a system (see figure 1). Interventions to catalyze system change will target one or more of the Rs. Capturing change at these points of intervention is more manageable and can usually be accomplished with familiar methods.
Lessons Learned
Over the past 20 years, SETIG has provided a home for collaboration, contestation and learning about how to understand and evaluate systems. In that time, we have clarified what systems informed evaluation is not. But within the community we continue to debate the contours of good practice. That work is taking on greater urgency as it becomes ever more evident that the biggest global challenges, like climate change, are crying out for system change.
The American Evaluation Association is hosting this week with our colleagues in the Systems in Evaluation Topical Interest Group. The contributions all this week to AEA365 come from SETIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.