Hello! I’m Emily Barkin, a public health program evaluator at Deloitte Consulting LLP. I design and evaluate training ranging from specific health care programs (e.g., women’s health) to workforce development (e.g., clinical informaticists, data managers) to large-scale health care operations (e.g., medical claims processing). While designing training across these various landscapes, I often question: Can we enhance the impact of the Kirkpatrick Model by using Program Evaluation frameworks to tell a better story of training?
The Kirkpatrick Model offers a grounded framework for designing training measures but does not overtly encourage goal setting and strategic thinking up front, prior to setting them up. Program Evaluation frameworks, such as CDC’s, inspire users to understand and create a story and then utilize measurement to back it up.
The Kirkpatrick Model is best known for evaluating training aptitude, based on four levels of assessment criteria, all building upon each other in succession.
Level 1: Reaction. Did participants enjoy and resonate with the training program? Would they recommend it to others? This may not link to outcomes but shows an increased likelihood future learners will pay attention to training content.
Level 2: Learning. What was learned? Were there any changes to expertise, knowledge, mindset, skills, attitudes?
Level 3: Transfer. Has participant behavior changed because of the training? Are skills from training being applied in practice?
Level 4: Results. What has been the overall success of training for the business/organization (e.g., quality, safety, efficiency, cost savings, production)? These measures are usually akin to ROI or Impact Evaluation with a control group.
The Kirkpatrick Model approach may imply, but does not directly encourage, user understanding of the training program (e.g., why it was launched, who the stakeholders and influencers are), understanding of stakeholder/user outcomes and goals, use of a theory of change (or logic model), and recommendations for continuous improvement, delivery of actionable recommendation, or continuous learning and dissemination planning. In essence, without the supplementation of Program Evaluation frameworks to the Kirkpatrick Model, the training itself can become isolated from the larger program ecosystem.
Some of the ways I’ve enhanced training measurement on my engagements is by continuing to use the Kirkpatrick Model to design training measures but bolstering it with elements of CDC’s Program Evaluation framework. This helps me and other training designers highlight direct linkages between short- and long-term data trends and outcomes, tell a better story, and deliver actionable results to stakeholders.
Rad Resources
Have you used the Kirkpatrick Model and/or Program Evaluation Frameworks to conduct training evaluation? Have you tried to merge the two? Or draw from a third framework around training measurement? What were the pros and cons of each? Let me know what your experience has been with evaluating training/learning programs!
The American Evaluation Association is hosting Health Evaluation TIG Week with our colleagues in the Health Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Health Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.