Hello, I am Rashell Bowerman, Educational Consultant for the Michigan Department of Education. I formally worked as a doctoral associate for the National Secondary Transition Technical Assistance Center (NSTTAC) and continue to participate in NSTTAC’s Evaluating Professional Development Community of Practice. NSTTAC is committed to data-based decision making and views evaluation as a tool for improving our work. In evaluating professional development, NSTTAC adopted a five level evaluation model looking across the continuum from participant reactions to student outcomes. The focus of this article is on Level One: participant reactions.
Lesson Learned:
- Good evaluations don’t have to be complicated. This is especially true when we look at collecting Level One data. Educators often do not pay much attention to evaluating their professional development activities. Many even consider evaluation to be too costly and time-consuming. Level one data is the simplest to gather and analyze, it is the most common form of professional development evaluation.
Terrific Tip:
- Looking at participant reactions gets at the initial thoughts and satisfaction regarding their experience. The following questions are typically addressed on a Level One evaluation:
- Did they like it?
- Was their time well spent?
- Did the material make sense?
- Will it be useful
- Was leader knowledgeable & helpful?
- Were the refreshments fresh & tasty?
- Was the room the right temp.?
- Were the chairs comfortable?
- By listening to the raw reaction of participants at the conclusion of the professional development activity, it can provide valuable information which can help improve the design and delivery of future programs or activities. Level on evaluation data can be gathered in several ways:
- Questionnaires administered at the end of the session
- Focus groups
- Interviews
- Personal learning logs
Radical Resources:
Here is an example of a section of an evaluation form used to collect level one data.
Other Rad Resources and Links:
- The Evaluation Exchange: A Conversation with Thomas Guskey
- Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press.
- NSTTAC Evaluation Toolkit
The American Evaluation Association is celebrating the Evaluating Professional Development Community of Practice (PDCoP) Week. The contributions all week come from PDCoP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.
Donald Kirkpatrick labeled these levels of evaluation over 50 years ago. There is still no evidence to my knowledge that measuring participant reactions to a professional development program tells us anything more than their immediate emotional response to the program. It doesn’t say anything about merit or worth, value or impact. It might be nice for the presenter to know if participants liked the program, but it doesn’t predict learning or application. Just thought we should be clear about this.
In Evaluating Professional Development (2000) Guskey fully acknowledges Kirkpatrick’s (as well as other) models in his work: “…it’s simplicity and practicality have made it the foundation of training program evaluations in businesses around the world” (p. 55), and shares that his own model “…represents an adaptation of the evaluation model advanced by Kirkpatrick” (p. 78). With regard to evaluating participants’ reactions, Guskey argues, “…measuring participants’ initial satisfaction with the experience provides information that can help improve the design and delivery of programs or activities in valid ways. In addition, positive reactions from participants are usually a necessary prerequisite to higher level evaluation results” (p. 82). He never advocates the use of just this one level, or to begin and end professional development evaluation at this level. What he does say is, “…success at one level is necessary for success at the levels that follow” (p. 78).
Very Nice. I wish there was a “like” button here.