Hi! I am Kelly Murphy, a Doctoral Candidate in Applied Developmental Psychology at Claremont Graduate University and a new Member-At-Large in the PreK-12 Educational Evaluation TIG. Over the past five years I have had the pleasure of working on an evaluation of a large multi-site out-of-school time (OST) program that serves over 15,000 K-12 students. Today I’m going to share some of the variables that I’ve come across that have improved my ability to sensitively measure program impact.
Hot Tip #1: We all know that sufficient participation in OST programs is essential for students to achieve desired outcomes, but what is the best way to measure student participation? While cumulative days attended is the most commonly used approach, I have consistently found very interesting effects when I include other indices of program participation such as duration of participation (number of months attended) and intensity of participation (ratio of days attended to days enrolled) in my analyses. By including multiple indices of program participation we can get a clearer picture of students’ attendance patterns and enhance our understanding of how participation relates to student outcomes.
Hot Tip #2: As OST programs are beginning to offer a wider array of activities to students (e.g., tutoring, performing arts, sports, and leadership) it is important to understand how participation in these different activities relates to outcomes.
By measuring attendance by activity type we can learn whether participation in different activities leads to differential outcomes in students and this information can help us better align our outcome measures to the specific contexts of our programs.
Hot Tip #3: Multi-site OST programs usually serve a fairly large and heterogeneous population of students that have the potential to “dilute” program effects. To overcome this issue it is important to disaggregate data by important student and site characteristics. Characteristics that we have found to be key moderators of program effect are school level, district association (i.e., public or charter), grade level, and the reason students joined the program (i.e., self-joined or other joined).
Rad Resource #1: The Harvard Family Research Project has free publications and resources for OST program evaluators.
The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.