Hi, my name is Stephanie Mooshegian, and I am an Assistant Professor of Organizational Studies at Saint Louis University. My doctoral studies were in Industrial/Organizational Psychology with a minor in Research Methods, and I am pleased to have been introduced to AEA.
Most of you are probably familiar with Kirkpatrick’s four level framework for program evaluation. Kirkpatrick suggested that programs can be evaluated for reaction (level 1), learning (level 2), application (level 3), and impact (level 4). While this language is applied most often to training program evaluation, the framework readily translates into educational settings, especially at the collegiate level. For example, student evaluations of teaching are distributed at the end of term (level 1). Grades are one indicator of learning (level 2), with some courses and degree programs emphasizing application (e.g., utilizing concepts in the real world) and impact (e.g., fulfilling a prerequisite; earning a degree or certificate).
While measures of learning, application, and impact contribute to a solid assessment of a course, I have found progress checks collected throughout the term (level 1), to be extremely beneficial. Progress checks are aimed at measuring affective reactions of students. When implemented on a regular basis, progress checks are a continuous feedback mechanism from which instructors can adjust to the class climate and facilitate learning most effectively.
Hot Tip: Progress checks can be applied in many learning formats. I have used checks in both traditional classroom and distance-learning formats (e.g., online learning, video-conferencing, webinars).
Cool Trick: Keep it simple. In traditional classrooms, it might be a show of hands for who is feeling tired, happy, exhausted, satisfied, or you could collect anonymous submissions of students current ‘state’ or feeling. In webinars, you might utilize emoticons or preprogrammed communication tools.
Cool Trick: Be creative! Ask students to stretch their imagination. I have used the prompt: “I feel like a (insert animal), because I am _____.” Some of my favorite responses were “turtle, because I am slow today” to “bird…ready to fly” to “frog…ready to jump into the next class.”
Cool Trick: Assess perceptions of course impact. For example, I have used Likert scale ratings to measure student satisfaction with the quality of their homework, term papers, and tests. Mid to late term, I have asked students to list the top three ways that they have applied concepts in class, and what they would like to learn before the end of the term.
Hot Tip: Engage students at the start of class. A progress check asking students to reflect on their current state at the start of class, helps them transition into the class meeting. In acknowledging that they are tired, hungry, excited, or content, the self-awareness assists in managing emotions appropriately.
This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.
Thanks for the contribution, Stephanie!
Stephanie, these are helpful suggestions on the use of Kirkpatrick’s Level 1 “reactionnaires” in traditional classroom and Web-based instruction. However, the research on the four levels has found very low correlation between them (in work settings). I just want to raise a caution that knowing how students feel about a course is not a good predictor of application or impact of their learning.