Hello! My name is Callie Dean, and I am a program evaluator with The Evaluation Group, where I work on several projects related to STEM and computer science education. Because of the overwhelming need to expand and diversify the STEM workforce, many organizations have developed initiatives for K-12 students with the goal of increasing awareness and interest in STEM careers.
In analyzing student survey results for a computer science education program, however, I noticed an unexpected trend: nearly all the STEM career-interest survey items saw decreases from the beginning to the end of the program. When I consulted with some colleagues, they mentioned they had observed similar trends in other evaluation projects.
What was going on? Were all these programs ineffective at achieving their desired outcomes, or was something else happening?
We looked a little deeper at existing literature and at our data, and we realized our program’s story might be a bit more complex than it appeared on the surface. Prior research shows that middle school is the prime time for students to hone their career interests. Our elementary program participants tended to have a broader range of interest in STEM careers than did the middle school program participants. In effect, the middle school students were honing their interests, becoming more interested in a smaller number of possible career options.
Now this was a story worth telling. Our clients ended up using this data to make some shifts in how they presented their career awareness curriculum to both teachers and students. We also developed some more specific survey questions to use in the future.
Hot Tips
Be clear about the outcomes you plan to measure. Career knowledge, awareness, and interest are related, but separate, outcomes. Before you start your project, know exactly which outcome (or combo of outcomes!) makes the most sense within your program’s context.
Use words that make sense for your program and participants. STEM is not a monolith: just because a student is interested in veterinary medicine does not mean they’ll also be interested in computer science! Make sure your instrument uses words and concepts that align with the specific programming you are evaluating and that will be familiar to your program participants.
Learn from your results. As our experience demonstrates, you may need to rethink your definition of success, incorporate findings from previous studies, and/or triangulate your results with other data sources about the program. Use your evaluation results formatively to better understand how your program works and build for future success!
Rad Resources
Here are a few career-interest surveys you can use or adapt for your next STEM evaluation project:
- The Student Attitudes Toward STEM (S-STEM) by the Friday Institute at NC State
- The STEM Career Interest Survey (STEM-CIS) by Kier et al.
- The STEM Semantics Survey by Knezek & Christensen
The American Evaluation Association is hosting STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to AEA365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
This is such an interesting article, Callie! You bring up several major considerations for affinity/interest surveys that I will have to incorporate into my own evaluations in the future. I especially appreciate the fact that you note the relation between students’ ages and where they are in their awareness of industries generally (e.g., STEM) versus careers specifically (e.g., computer science). It makes sense that we cannot expect 100% of students in mass-enrollment STEM programs to ultimately be interested in STEM and/or pursue STEM. Perhaps there is another measure (or measures) that should be evaluated, such as the percent of students who show growth and the degree to which they show growth.
Thanks for your perspective!