ECLIPS and Systems TIG Week: Tarek Azzam and Matt Keene Recap Systems-Oriented Evaluation

Hello! We are Tarek Azzam (Claremont Graduate University) and Matt Keene (Environmental Protection Agency). We are members of the External Review Panel for ECLIPS.

Whoa! Even though it’s oh-so-tempting to try, you don’t need the perfect string of words to define “systems-oriented evaluation.”

Lesson Learned: It’s in the roots of evaluation 

At its core a systems approach to evaluation encourages the evaluator to consider the physical, political, and structural issues that surround a program, and to examine how these factors help or hinder the success of a program. This examination and reflection process is incorporated in the work and writings of lead evaluation scholars such as Lee Cronbach, Robert Stake, and Jennifer Greene. The presence of systems thinking also can be seen in our standards (specifically standards F3, A4, and A7).

And it’s also something different, because it requires the evaluator to recognize that the program is part of systems that have their own dynamics. It forces the evaluation to examine issues that go beyond the process and outcomes of a single program.

Lesson Learned: How to become a systems-oriented evaluator

1)      Adopt some habits of systems thinkers…

2)     Know the domains of Social Ecology and use them to understand the leverage points of boundaries, relationships, and perspectives. Donella Meadows says that boundaries are problem dependent and messy. Don’t make the world linear for your mathematical or administrative convenience.

3)     Delve into the dynamics of systems to find the regions of organized, adaptive, and unorganized patterns.

4)     Find leverage points (places to intervene where small tweaks can lead to big changes). Here are some leverage points from Meadows you can use to influence relevant systems:]

5)     Let systems thinking do fuzzy things to your logic model. A fuzzy logic model takes into account the dynamic nature of the systems surrounding a program. It gives a visual image of the complexities that can affect processes and outcomes.

In the ECLIPS, all of our logic models used to look like this:

But after applying systems thinking, we made them into fuzzy logic models. See how different they look.

Try creating a fuzzy logic model to find and depict the system’s complexity, making your logic model more useful to people for a longer time.

That’s all for now

This concludes ECLIPS week on aea365! Don’t expect to learn about systems alone or in a short period of time. It may well be a journey into a new way of thinking about evaluation. Get involved in an existing community of practice about systems or form your own group. ECLIPS members are happy to share what they are doing.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.