Hello all- we are Laura Weiss and Deborah Wasserman, researchers with COSI’s Center for Research and Evaluation and want to share with you an exciting tool for facilitating co-created informal STEM educational program planning and evaluation.
Informal STEM learning institutions, educators, and evaluators face a unique challenge. Each institution sponsors a wide array of programming for all sorts of learners. Educational interactions can range from two- or three-minute interactions to multi-year intensive programming. Across a broad spectrum of programming contact time, and learners, how do we plan and evaluate these programs in a way that is useful to the program providers while contributing to a single understanding of institutional impact?
Rad Resources
Collaboratively with the National Museum of Natural History (NMNH), and the US Air & Space Force K-12 Outreach, we have developed the ISEE (Informal STEM Education and Evaluation) Strategic Outcomes Framework, which addresses these challenges. After incorporating recommendations and feedback from educators, practitioners, and content knowledge scholars gathered in an NSF-funded conference (#2039209), the framework is becoming a useful tool. The Strategic Outcomes Framework builds on accepted informal learning categories and strands (Bell et al., 2009; Friedman, 2008) by delineating realistic outcome expectations based on dosage and complexity of programming. Specifically, the Framework’s outcomes exist within eight categories: interest, attitude, knowledge, STEM skills, 21st century skills & social emotional development and learning (SED/L), behavior, STEM capital, and career building. Each category includes six outcome types that can be mapped to programming that ranges from short duration with simple content to extended duration with complex content.
Lessons Learned
Selection and application of these outcomes involves some key principles and guidelines and generates important dialog. First, a program rarely targets a single outcome. Instead, program design derives from outcome combinations, each of which leads to different educational strategies. These combinations can occur both between and within categories. Programs can use the Framework to select primary and supporting outcomes. For example, a program with the primary outcome of creating a behavior change may have supporting attitude, knowledge, and skill outcomes that participants need to be able to make that change. A program also may be defined in terms of expected and “reach” outcomes. For example, all participants in a workshop may be expected to report that they experienced situational interest, and the workshop also has the goal that some of those participants will develop personal interest.
Moreover, outcome types and even depth levels are value free. A program that targets a simple, less complex outcome can be equally valuable as one that targets deeper, more complex outcomes. Second, outcome depth does not necessarily correlate with developmental age or learner academic standing. Conceptually, given the time and creative programming strategy, some level six outcomes can be targeted even in preschool settings.
Another principle is that the Framework has been designed to nest within a system that acknowledges the voices of value holders (aka stakeholders) while selecting, defining, and strategizing toward achieving these outcomes. We have nested the Framework within what we call “Voices and Choices Planning for Self Determination,” a process, still in development, meant to facilitate co-creation, equalize power, and include a full diversity of value-holder voices at the table. With this process, varied voices contribute not only to outcome selection, but also to outcome definition, implementation strategies, and outcome measurement.
Get Involved
We continue to develop and pilot test this Framework, learning more with each collaborator. Contact us and be part of a conversation about how to use the Framework.
The American Evaluation Association is hosting STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to AEA365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.