Hi, we are Tim Sheldon and Jane Fields, Research Associates at the Center for Applied Research and Educational Improvement (CAREI) at the University of Minnesota. We serve as external evaluators for EngrTEAMS, a five-year, $8 million project funded by the National Science Foundation. The project is a partnership involving the University of Minnesota’s Science, Technology, Engineering, and Mathematics Education Center (the STEM Center) and Center for Compact and Efficient Fluid Power (CCEFP); Purdue University’s Institute for P-12 Engineering Research and Learning (INSPIRE!); and several school districts. EngrTEAMS is designed to increase students’ learning of science content, as well as mathematical concepts related to data analysis and measurement, by using an engineering design-based approach to teacher professional development and curriculum development.
As the external evaluators for this project, we based our evaluation framework on Guskey’s five levels of professional development (PD) evaluation (Guskey, 2002). He suggests evaluating (1) participant perceptions of the PD; (2) the knowledge and skills gained by participants; (3) the support from, and impact on, the organization; (4) participants use of their new knowledge and skills; and (5) the impact on student outcomes. In Guskey’s model, the aspects to be evaluated begin after delivery of the PD; that is, the framework does not specifically suggest assessing differences in participants or organizations prior to the delivery of the PD.
In the case of EngrTEAMS and other PD we have evaluated, we have noticed that even though participants receive the same training (i.e., the same “treatment”), their capacity to apply the new knowledge and skills (Guskey level 4) is not the same. What might explain this? We suggest that one way to better understand and explain these differences in implementation (and eventually student outcomes) is to also better understand participants and their organizations prior to the PD. Not all participants start the PD in the same place; for example, participants come to the PD with different levels of prior knowledge, different attitudes about the PD, different classroom management abilities, and different levels of organizational support.
When possible, assess implementation readiness of participants and their organizations prior to the delivery of the PD. This may include obtaining information about organizational readiness to support novel approaches, as well as participants’ prior content knowledge and classroom experience, their perception of school or district buy-in, and participants’ attitudes about the training and future adoption of what they will be learning.
- Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45-51.
- Guskey, T. R. (2009). Closing the Knowledge Gap on Effective Professional Development. Educational Horizons, 87(4), 224-233 (http://files.eric.ed.gov/fulltext/EJ849021.pdf).
- Information about EngrTEAMS: http://engrteams.mspnet.org/ and https://sites.google.com/a/umn.edu/engrteams/.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.