Allow me to introduce myself as Anane Olatunji, president of Align Education, LLC, a consulting R & D firm. Having worked with all types of educational agencies over the last two decades, I’d like to share one important tip that I’ve found particularly helpful when evaluating educational program evaluations. Assess student engagement!
Although there is no agreed upon definition among researchers for the term student engagement, it has to do with the quality of students’ involvement in school based on their behaviors and feelings or attitudes (see Yazzie-Mintz and McCormick, 2012). To underscore the need for assessing engagement, I’d like to borrow a line from a document recently used in my work on a state-level evaluation of charter schools. A Report from the National Consensus Panel on Charter School Academic Quality contends that student engagement is “a precondition essential for achieving other educational outcomes.” In other words, engagement is a bellwether of academic achievement, the critical outcome educational concern. Whether engagement is high or low, achievement usually follows in the same direction. This information thus enables a program to make modifications, if needed, prior to summative evaluation. It is precisely for this reason that assessing engagement adds value to program evaluations. Here’s a simplified illustration of the role of engagement:
Unfortunately, even though engagement is an antecedent of achievement, it often is not assessed in evaluations. This omission may in part be due to program managers rather than evaluators. If managers don’t explicitly express an interest in assessing engagement, we as evaluators may be inclined to leave it at that and not push any further. My hope, however, is that you will take “program evaluation destiny” into your own hands. Through your awareness and use of this knowledge, you can improve quality of not only an evaluation, but also and more importantly – an educational program as a whole.
So how do you move from knowledge to implementation? Student attendance is one of the most common measures engagement. A shortcoming of this indicator, however, is that it doesn’t give a good indication about why students go to school. If most kids goes to school because the law or their parents force them to, then attendance alone can be a poor measure of engagement. Other measures therefore might include tardiness rates, rates of participation in school activities, or student satisfaction rates. For examples of survey items, see national surveys of middle and secondary school students. It’s especially important to assess at these levels because engagement declines after elementary school.
Of course, we’ve only scratched the surface on the topic of assessing engagement, but at least now you can move begin moving forward better than before. Good luck!
The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Thank you so much Anane for this post! You are so right, engagement must be considered when evaluating an educational program. There is a strong link between student engagement and student learning, yet as you said, we so often forget to evaluate engagement as one of our variables.
When you wrote, “Student attendance is one of the most common measures engagement.” I instantly remember several situations around student attendance. Students who had perfect on time attendance during a drama troupe artist in residence program, but horrible attendance the rest of the year. Or a student who went from poor attendance in elementary to almost perfect attendance at a sports school for junior high. We need to ensure that programs that engage students and their learning succeed!
Thank you, Anane, for highlighting the importance of assessing student engagement as a part of our evaluations. I can see how this approach could give evaluators and educators a more nuanced understanding of the factors that lead to program success. Would you agree that, in evaluating student engagement, we are focussing on the quality of not only program outputs (i.e. student achievement) but also program inputs (i.e. engagement and motivation)? It seems to me that this is a good example of how a Social Science theory-based approach to evaluation can lead to more fulsome and useful findings.
Best,
Ann-Marie
Dear Anane Olatunji,
It was so refreshing to read your article on student engagement. I am a teacher and currently a Masters of Education student studying evaluation. The connection between student engagement and success is an important one and like you mentioned I feel it is often overlooked. As an evaluator how do you coach your stakeholders or intended users to include such elements? Do you feel there is a difference between participation and engagement? If so, how would you make this distinction in your evaluation? When considering student engagement do you feel like external elements such as the time of day would come into play? I feel the start of engagement could come from giving students agency over their own learning and coaching them to move from seeking grades to seeking knowledge. Once this happens possible we will see a higher engagement rate in programs. At times I feel like so much evaluation is summative information and lose the importance of formative, especially with engagement.
Thanks for considering, Lorraine
Hello, Anane Olatunji.
Thank you for shedding light on the necessary topic of student engagement when evaluating a program. I find it imperative to measure the effectiveness of a program based on outcomes but ALSO on engagement. Our students are more likely to benefit from a program they are engaged in. If we are truly going to impact their lives through their participation in a program do we not want them to be engaged? It is unfortunate that many measures of programs are quantitative measures that are not inclusive of engagement. I agree that it would be helpful to take the evaluation destiny into our own hands and include valuable data on engagement. A program is not worth much if it is not engaging. Student engagement increases achievement. That is documented. Therefore, engagement in programs should increase positive outcomes.
Have you had found any more appropriate measures of engagement past attendance and tardiness that would be appropriate measures? I would be curious to add this information in the evaluation of a program we are implementing in a grade 5-9 school. Attendance is not an appropriate measure in our school because we are a rural school where students are bussed and it is not possible to “skip” school as there is nowhere to go.
I look forward to hearing back from you.
Thank you for your time.
Shelley
Dear Anane,
Thank you for your post on the importance of evaluating for student engagement. As you identified, there is a direct causal link between engagement and achievement and more students would benefit if teachers and school leaders evaluated the level of student engagement and responded promptly to the data they collected.
I fully agree that including student engagement in programme evaluation would improve the quality of the evaluation; it would also improve the usability of it. Teachers who are made sensitive to how deeply their students are engaged in their learning, would be much better equipped to respond to a lack of engagement in a timely manner. By going through a process of regularly collecting data on student engagement, most teachers would feel compelled to make changes in their teaching practice before these practices became too ingrained and part of “how we do things”.
Including the collection of this data, particularly as part of a process evaluation, would certainly help stakeholders identify problems and correct them before investing more resources into a programme and losing more instructional time.
Anane – you’re so right! As a parent of teens I can’t underscore the need for this (and teacher/ school adjustments needed) as a result of such data.
As an evaluator, we need such engagement data in int’l development… that’s a precursor to capacity built and motivation needed to sustain projects after we leave.
Thanks so much,
Jindra