Hello – we are a team of evaluators and researchers who work together on a project called Streamlinine Embedded Assessments (NSF #1713434), and we want to share a bit about our efforts to develop embedded assessments as a form of shared measures that can serve the needs and context within citizen science and other forms of science learning. Our team brings extensive experience in research and evaluation, and is based in non-profits, universities and museums: Rachel Becker-Klein, Veronica Del Bianco, Amy Grack Nelson, Jenna Linhart, Karen Peterman, Tina Phillips, Cathlyn (Cat) Davis Stylinski, and Andrea Wiggins. We are partnering directly with practitioners from 10 citizen science projects across the U.S. on this exploratory research.
First, let’s clarify some terms.
- Shared measures are grounded in theory and developed with psychometrics to support their use across contexts. They should be broadly applicable.
- Embedded assessments (EAs) are activities that are integrated into the learning experience, allowing learners to demonstrate their competencies. They should be performance-based, authentic, and embedded in program activities (of course!).
- Citizen science has participants apply science inquiry skills to contribute to research endeavors.
Previous research demonstrates that formalized skill assessment of any type is uncommon in citizen science. EAs offer tremendous potential. So, the challenge is how do we create shared measures of volunteer skills that meet our criteria: (1) broadly applicable to multiple projects, (2) indistinguishable from the learning experience itself, (3) performance-based with learners demonstrating their proficiency, and (4) authentic to the learning experience.
Lessons Learned
As we work with our partnering citizen science project leaders, we are finding that there are tensions among the four criteria, and it may not be possible to fulfill all criteria equally. Embedded and authentic tend to complement each other. However, the more embedded an assessment becomes the less broadly applicable it would likely be to other settings.
For instance, we created a simulation video of a volunteer collecting data in the field. It was performance-based because it had volunteers demonstrate competencies, not rate them, but watching a video lacks the authenticity of actually being outside collecting the data themselves.
Rad Resources:
Here are some resources that we are using as we continue our work on creating shared measures:
- Our definition for shared measures comes from this article: Shared Measures for Evaluating Common Outcomes of Informal STEM Education Experiences in New Directions for Evaluation March 2019
- The DEVISE scales measure a broad range of constructs related to citizen science and complement the kinds of EAs we have developed to date
- The EvalFest project includes shared measures for science festivals, as well as training videos for how to collect data
The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the Science, Technology, Engineering, and Mathematics Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM Education and Training TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.