Organizations (business, non-profits, and government) are paying more attention to the impact of employee training and learning programs. They recognize that in this Knowledge Economy, learning and change must be a core competency. I’m Stephen J. Gill and I have been evaluating the impact of learning interventions in organizations for the past 35 years. Recently, LAD Global, an affiliate of the Singapore Training and Development Association, asked me to record a free, online, short course for them titled, Measuring and Evaluating Learning in the Knowledge Economy. In that course, I summarize lessons I’ve learned from the evaluation of many different learning interventions in organizations. I cover the following topics in the course:
- Introduction to Measurement and Evaluation of Learning
- Purpose of Measurement and Evaluation
- Methods of Measurement and Evaluation
- Process of Training, Learning and Achieving Results
- Designing Measurement and Evaluation
- Using Evaluation Results to Make a Difference
These topics are addressed over 72 minutes broken up into 6 modules. The course is designed to be an introductory look at the issues and practices that confront evaluators who want to enhance the impact of employee learning and development programs in their organizations.
The course covers the purpose of measuring and evaluating learning in the Knowledge Economy, some practical information about why evaluation is important in organizations, the different methods of measurement, how proficient practitioners need to be with technology and analytics, the best ways of presenting information to stakeholders, why learners and their managers need to be held accountable for results, how to approach designing measurement and evaluation for your organization, and using the evaluation results to make a difference in your organization.
Lessons Learned:
- Engage stakeholders (learners, their managers, executives, team leaders, etc.) in designing evaluation of learning interventions.
- Emphasis should be on improving the learning intervention, not proving that it works or not.
- Do not use evaluation findings to find fault with people.
- Match the evaluation method with the questions you are trying to answer. Kirkpatrick’s Four Levels, Phillip’s ROI, and Brinkerhoff’s Success Case Method are three commonly used options.
- Engage stakeholders in reflecting on the meaning of evaluation findings and the implications for their work.
- Achieving positive results from a learning intervention has as much if not more to do with the culture of the organization and the support from managers. Evaluate those elements.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Dr. Gill,
Thank you for the reminder of the value in process over outcome in your “Lessons Learned”. Although I am new to evaluation, from my limited experience, the emphasis on findings over the learning process seems to have dominated evaluation literature in the past and it is exciting to see the change in emphasis more recently. As an elementary school teacher, I see the value in students recognizing their own learning growth, learning how to learn, and being able to articulate their growth. No matter what area of employment – business, technology, services – those who are able to learn and be flexible and adaptable to change have an advantage.
When evaluating learning in a particular working environment, Kirkpatrick mentions that the third level of evaluation is transfer (Kirkpatrick, 2013). I am wondering if you have an example of how you have assessed whether or not workers are using the acquired knowledge from the program in their everyday environment? This certainly would reflect true effectiveness of a program, but at what point in the evaluation do you measure when the change in behavior has occurred?
Also, you mentioned that the culture of the organization and support from managers plays an important role in achieving more positive results. In your experience, have you found that managers who are more open to mistake-making have more risk-takers and higher rates of learning among staff?
Thank you for your post!
Laura
Kirkpatrick, Dr. Donald, and Kirkpatrick, Dr. Jim (2013). Kirkpatrick Four Levels Audio Recording Guide. Retrieved August 12, 2018 from: https://www.kirkpatrickpartners.com/Portals/0/Products/Kirkpatrick%20Four%20Levels%20-%20Audio%20Recordings%20Study%20Guide.pdf
This is great information and as a new evaluator, these moduels will be of significant advancement. Thank you Dr. Gill!