ECLIPS & Systems TIG Week: David Hata on Looking to Interconnections for Value Creation

Hello! I’m David Hata, an independent consultant who serves as an external evaluator for a number of NSF-funded Advanced Technological Education (ATE) projects and centers.

I’m here to share my experience helping a STEM project see itself as part of a system that creates value. For some clients, success is doing a lot of activities. The clients may not be thinking about what value those activities create.

My early evaluations focused on evaluating individual activities using methodologies such as described in Kirkpatrick’ book Evaluating Training Programs: Four Levels. With my recent participation in ECLIPS and the Synergy Project, I have started to think of evaluation more holistically in terms of systems and value creation. I now view NSF ATE projects and centers as systems that create value at multiple places within their structure. By recognizing these value-creation systems, I am discovering more ways that evaluation can help fulfill the mission of the ATE program at NSF—to increase the number of and strengthen skills of technicians for the workforce through the implementation of workforce development initiatives.

Lesson Learned: Finding Value in the Interconnections  

A useful conceptual framework for assessing value creation has been developed by Etienne Wenger and colleagues.

They outline five types of value: immediate, potential, applied, realized, and reframing. I used the first four types of value to help develop a road map for The Southwest Center for Microsystems Education (SCME), an ATE regional center at the University of New Mexico in Albuquerque. Working with my client to create the road map helped them develop a systems view of their project. The road map shows the connections between grant activities and a career pathway that produces advanced technicians for the U.S. workforce.

The diagram below shows how SCME activities fit into a career pathway from high school to community college to workforce.

Center activities can be viewed as value investments:

• immediate value: knowledge, skills, and materials gained by each participant;
• potential value: what each teacher plans to do with their new knowledge, skills, and materials;
• applied value: what changed in classroom instruction and student learning;
• realized value: number of microsystem technicians produced based on graduate data.

The evaluation measures the value created by these investments at different points in the system. The links in the diagram emphasize the nature of the exchanges. Understanding the links between the boxes is as important as defining the activities and outcomes in the boxes.

Check out the following resource and join us tomorrow as my fellow ECLIPS members examine system patterns via a Robot diagram.

Rad Resource: Donella Meadows’ book, Thinking in Systems.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.