Hello, we are David Reider (Education Design, Inc.), Ginger Fitzhugh (Evaluation and Research Associates), and Alyssa Na’im (Education Development Center, Inc.). As ECLIPS members, we are incorporating systems concepts into STEM education evaluations related to the National Science Foundation program, Innovative Technology Experiences for Students and Teachers (ITEST).
Keying off the iceberg diagram (see Monday’s post), we go deeper into a system to find leverage points for change by considering:
- boundaries (demarcations that define regions or entities)
- relationships (connections and exchanges between project parts or people)
- perspectives (the paradigms held by various parties and the purposes they seek)
Hot Tip: Don’t disregard the simple in a complex setting.
I (David) am evaluating a project using science probes and models in K-12 classrooms in four states. Although there were vastly different support structures in the sites, one of the lessons I learned in my evaluation was quite simple. The more frequently teachers posted to the online learning platform (thus reaching beyond the boundaries of their classroom), the higher the quality of their classroom projects. This was a case of leveraging small actions toward larger gains.
Hot Tip: Ask about boundaries, relationships and perspectives.
Questions That Matter has terrific examples of evaluation and interview questions that relate to boundaries, relationships and perspectives. I (Ginger) added several of these questions to our interview protocols for project leaders. For example, we added, “What, if any, unanticipated outcomes (positive or negative) have happened in the project thus far?” We learned that parents were interested in obtaining the project equipment to use with their children. This spurred the project team to consider how to make the materials more widely available.
Hot Tip: Consider how program goals can be leveraged.
I (Alyssa) am now paying more attention to acknowledging and identifying the boundaries, perspectives, and relationships for both program implementation and evaluation. Programs express their perspective through their statement of purpose, e.g., improving the nation’s STEM workforce development capacity. Systems thinking helps us understand why their strategies to accomplish this purpose may overlap or diverge and to see possible leverage points for change.
Hot Tip: Don’t lead with systems language; find ways to include it.
It’s not always necessary to say, “This is a systems idea.” Rather, use familiar language to explain how what you are doing adds value to the evaluation and the project.
Join us again tomorrow as we move away from the Iceberg diagram to consider the role of evaluators.
Rad Resources
- Ginger’s 2012 AEA presentation, Challenges and Successes Associated with Introducing Systems Concepts to an Existing Evaluation, is about applying boundaries, relationships, and perspectives to evaluations.
- David’s 2012 AEA presentation, Leverage Points: Places to Intervene in a System.
- ITEST LRC website
The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.