Hi! We are Seth Kaplan and Sue Ann Sarpy. Seth is an assistant professor of Industrial/Organizational Psychology at George Mason University, and Sue Ann is an assistant Principal of Sarpy and Associates, LLC and Clinical Assistant Professor of Environmental Health Sciences at Tulane University School of Public Health and Tropical Medicine. Today we will be sharing a few tips about considering organizational factors when trying to evaluate training initiatives.
In examining the effects of training programs in improving on-the-job performance, it should be noted that positive learning gains do not necessarily translate into the participants performing the training-related behaviors once back on the job. A primary reason for this disconnect is the influence of contextual or organizational factors that facilitate or impede the attainment of high levels of effectiveness. In effect, these conditions place an upper limit on the potential impact that the training may have on related outcomes.
Common organizational factors, which can either enhance or hinder job behavior, including training-related behavior, include: time and scheduling issues, budgetary and financial considerations, the presence or absence of relevant supplies and materials, the adequacy of the physical work environment, the competence and cooperation of other organizational members and partners, and the extent to which the job context mirrors the training context and provides opportunity for practicing training-related behaviors.
Hot Tip: We strongly recommend that evaluation of training programs should include a systematic assessment of these organizational factors to examine their role in hindering or facilitating the transference of training-related behaviors. Identification of organizational barriers can serve to explain why particular aspects of the training content did not translate into the expected increases in on-the-job effectiveness. Also, after learning which factors are most frequent and problematic, the training program developers can incorporate this information into subsequent iterations of the training by alerting trainees of these barriers and by teaching them skills to address and overcome them. In addition, the program developers can leverage the organizational facilitators by emphasizing the importance of these factors in producing superior performance.
Hot Tip: Evaluation of these factors typically is done by constructing a standard survey on which trainees assess the extent to which each factor facilitates or impedes their on-the-job effectiveness (e.g., on a 7-point Likert-type scale). Subject matter experts should provide the factors that will appear in the survey. In our experience though, the factors are rather consistent across occupations and industries. In order to increase the reliability of these ratings, we would recommend also surveying other individuals who work in the trainees’ office or department. Ideally, there should be convergence among these responses. Mean values then can be computed to determine which factors serve as the most influential barriers and facilitators.
This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.
I think it’s great that you are bringing attention to organizational factors that should be considered in any evaluation of the impact of training. Along the same lines, I use the “5As Framework”, which I developed with Sean Murray. This framework overlaps with some of your factors and has some additional factors that need attention when evaluating training (or any change intervention, for that matter). These are 1) alignment, 2) anticipation, 3) alliance, 4) application, and 5)accountability. You can read more about this Framework at http://www.realtimeperformance/ebook