I’m Amy Gaumer Erickson, Ph.D., and I’m an assistant research professor at the University of Kansas. My work focuses on the evaluation of effective instructional practices within MTSS (multi-tiered system of supports) educational models. I’ve found that though we expect educators to implement interventions with fidelity, we don’t always evaluate the professional development provided to educators.
Hot Tip: Educational evaluation must ensure that professional development is of high quality prior to holding educators responsible for the fidelity of implementation.
To evaluate the quality of training, the U.S. Department of Education Office of Special Education Programs has identified five indicators based on implementation science research:
- Accountability for delivery and quality monitoring of training is clear.
- Adult learning principles are used.
- Training is skill-based.
- Outcome data is collected and analyzed (pre- and post-testing) to assess participants’ knowledge and skills.
- Trainers are trained, coached, and observed. Data are used to improve trainer skills and the content of training.
External evaluators on State Personnel Development Grants use these indicators to guide their evaluation of training. In practice, my colleagues and I have observed numerous trainings and workshops to evaluate these trainings’ alignment to the projects’ learning objectives and assess trainers’ use of adult learning principles and participant rehearsals. We’ve also supported trainers in their development and deployment of knowledge-based pre-/post-assessments. Data collected from the assessments and observations provides trainers with evidence of their effectiveness and is used to coach them in areas in which they could improve. The result is higher training fidelity.
After observing educational trainings across a variety of content areas and conducting a review of research-identified indicators of effective professional development, we developed an observation tool that evaluates the use of adult learning principles through training practices relating to demonstration, engagement, evaluation, and mastery. This tool has supported trainers in developing their content and continuously improving training. Within train-the-trainer models, this tool has been used alongside content-specific pre- and post-testing to ensure that training fidelity is maintained throughout the scale-up process.
Rad Resource: Download the Observation Checklist for High-Quality Professional Development Training, and use it to observe the quality of the professional development provided to educators.
To learn more about the development and implementation of the Observation Checklist for High-Quality Professional Development Training, watch our AEA Ignite Session.
The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Dr. Ericsson,
I have read your article on Evaluating the Quality of Professional Development and I have found it very enlightening what strikes me are the five indicators provided in the blog. The checklist you have provided is also very practical which makes the whole process simple and easy to implement. I am curious to know if there are other criteria which can be used in evaluating the quality of professional development and if so are there any conditions of these additional criteria.
Such criteria which can be included such as feedback from the teachers, not immediate feedback but information provided to determine if the teachers have applied what was learned during the professional development session. Even though such a criterium would be heavily dependent on other factors such as the environment and personality of the individual. One way to determine if a student has learned anything would be to apply what the student has been taught. Could the same be applied to teachers for professional development seminars concerning what they have learned? Teachers applying the knowledge (if applicable) gained by these seminars and what other criteria can and cannot be used when evaluating the quality of professional development?
Regards,
Kerrin Celestine
Yes, there are definitely other criteria to be considered in the evaluation of professional development. The checklist only focuses on the workshop/training element. While behavior rehearsals or role play can be incorporated into training, ongoing job-embedded coaching is ideal for the transfer of learning into practice. The coaching process and impact of coaching is then evaluated.
When coaching is not available, we have used implementation self-assessments that ask participants which practices they’ve used in their classrooms and how these practices have impacted students. Such self-assessments are usually sent out 3 weeks to 2 months after the training, depending on the practices taught. The self-assessment provides opportunities for the educators to reflect on the training and their implementation, as well as inform the trainer of additional resources or support that would be beneficial.
In some situations, trainers give “homework” to all participants. This typically is in the form of an action plan with detailed action steps. The trainer then keeps a copy of each person’s or team’s action plan and follows up on the completion of each action step.
I’ve also seen trainings for teachers that include information that is sent to the school principals, outlining the instructional practices taught and what principals can look for during classroom walk-throughs. I haven’t yet had the opportunity to evaluate the impact of this implementation strategy.
For a discussion of professional development features that are associated with positive outcomes, see the following review article:
Lauer, P. A., & Christopher, D. (2013). The impact of short-term professional development on participant outcomes: A review of the literature. Professional Development in Education, 40(2), 207-227.
http://www.tandfonline.com/doi/full/10.1080/.U3Zty_ldWSo#.U3Zv6vldWSo
Marty, good point. That’s why I’m a big proponent of using rubrics which show the gradations of practice (e.g., based on the CBAM model). Videos of practice further add to the development of what Kintsch (1988, 1998) calls the situation model.
Chad, thanks for your thoughts. We’ve used the same rubric for observing PD as for the classroom observation – with obvious adaptations in language. We find this tracks most closely to the elements and we can closely follow one or another across implementations.
Whatever works for an evaluator in either case is certainly better than not tracking PD at all!
Best.
Marty
Amy, this is a nice list of qualities. Thanks for putting them together. There is one piece missing that we have found critical to classroom implementation with fidelity – which may be there but is not obvious to me. That piece is that the PD professionals model what they want the teachers to do in the classroom. To talk about best practice, cite research, and let teachers practice is, of course, strong. However, if they are not seeing in the PD staff what and how they should be teaching, the chance of transfer to the classroom is not high.
We have compared quality of PD and extent of implementation of inquiry in the PD to extent of implementation in the classroom and find they match almost perfectly. If a presenter takes the inquiry model to the last phase then talks about what they should be doing and seeing in the last phase without the teachers experiencing it, that’s exactly what happens in the classroom. The teacher takes it to the last phase then talks about the right answer.
We’d recommend you consider adding a section on modeling whatever you are promoting within the PD so that teachers can see the entire process from start to finish, and, more importantly, experience what the students might be feeling/experiencing during the process in their classroom.
Thanks for pulling this list together.
Marty Henry
Marty, thank you for your comments. I agree that modeling or illustrating the practice is important. In this observation we address it to an extent through items 12 and 13. Much of the PD we evaluate is not on specific instructional strategies, so in those cases, we find that PD providers often show videos of the practice in action, but cannot directly model it during the training. We use a coaching observation to evaluate the quality of ongoing coaching that supports teachers’ implementation of evidence-based instructional practices.
Could you please share what could not be modeled in a PD experience. I’m drawing a blank.
Thanks,
Marty