Hi, we are Tim Sheldon and Jane Fields, Research Associates at the Center for Applied Research and Educational Improvement (CAREI) at the University of Minnesota. We serve as external evaluators for EngrTEAMS, a five-year, $8 million project funded by the National Science Foundation. The project is a partnership involving the University of Minnesota’s Science, Technology, Engineering, and Mathematics Education Center (the STEM Center) and Center for Compact and Efficient Fluid Power (CCEFP); Purdue University’s Institute for P-12 Engineering Research and Learning (INSPIRE!); and several school districts. EngrTEAMS is designed to increase students’ learning of science content, as well as mathematical concepts related to data analysis and measurement, by using an engineering design-based approach to teacher professional development and curriculum development.
Context:
As the external evaluators for this project, we based our evaluation framework on Guskey’s five levels of professional development (PD) evaluation (Guskey, 2002). He suggests evaluating (1) participant perceptions of the PD; (2) the knowledge and skills gained by participants; (3) the support from, and impact on, the organization; (4) participants use of their new knowledge and skills; and (5) the impact on student outcomes. In Guskey’s model, the aspects to be evaluated begin after delivery of the PD; that is, the framework does not specifically suggest assessing differences in participants or organizations prior to the delivery of the PD.
In the case of EngrTEAMS and other PD we have evaluated, we have noticed that even though participants receive the same training (i.e., the same “treatment”), their capacity to apply the new knowledge and skills (Guskey level 4) is not the same. What might explain this? We suggest that one way to better understand and explain these differences in implementation (and eventually student outcomes) is to also better understand participants and their organizations prior to the PD. Not all participants start the PD in the same place; for example, participants come to the PD with different levels of prior knowledge, different attitudes about the PD, different classroom management abilities, and different levels of organizational support.
Lesson learned:
When possible, assess implementation readiness of participants and their organizations prior to the delivery of the PD. This may include obtaining information about organizational readiness to support novel approaches, as well as participants’ prior content knowledge and classroom experience, their perception of school or district buy-in, and participants’ attitudes about the training and future adoption of what they will be learning.
Rad Resources:
- Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45-51.
- Guskey, T. R. (2009). Closing the Knowledge Gap on Effective Professional Development. Educational Horizons, 87(4), 224-233 (http://files.eric.ed.gov/fulltext/EJ849021.pdf).
- Information about EngrTEAMS: http://engrteams.mspnet.org/ and https://sites.google.com/a/umn.edu/engrteams/.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello! As an elementary school Principal, I am responsible for delivering Professional Learning to my staff of 75 teachers and for evaluating whether the PL session has been effective. In the language of Guskey’s model, assessing levels four and five is both simple and complicated. On a simple level, I can assess through learning walks, work on student desks, professional conversations with my staff, and agreed-upon achievement metrics, whether the PL is having a desired effect. When everything is trending in the right direction, everyone feels satisfied with a job well done.
Unfortunately, assessing the impact of Professional Learning is never quite that simple. I have often wondered about why some staff are more willing, able, and successful at giving new teaching strategies a go, especially when the effectiveness of the new approach is well documented and supported.
I find your discussion of a “readiness” evaluation to be helpful – perhaps getting a better sense of where staff are at in their professional knowledge, not to mention in their mindset (growth vs. fixed), their biases, and their self-perception as professionals, will gain me more buy-in and better traction with my PL sessions.
Do you believe that there are readiness evaluation tools available with any sort of generalizable impact? I woud love to implement some sort of readiness assessment without feeling like I have to reinvent the wheel. Would you be able to share some of the readiness assessment tools that you recommend?
Thank you!
Dave Morgan