Greetings from the Midwest! We are Jenell Holstead (University of Wisconsin-Green Bay) and Mindy Hightower King (Indiana University). For the past eight years, we have evaluated 21st Century Community Learning after school programs both at the local level, as well as statewide initiatives in Indiana and Kentucky. This post discusses best practices for evaluating out-of-school time programs intended to boost academic achievement.
Hot Tips for an Effective Out-of-School Time Program Evaluation
Hot Tip: Importance of “Point of Service” Practices. External evaluation of out-of-school time programs often focus on outcomes such as grades, test scores, and survey data. Such evaluation often place less emphasis on the practices that influence the after school environment and the program activities youth experience—the “point of service” (POS) aspects of after school quality. However, after school program staff can utilize evaluations of POS practices to systematically review the quality of their efforts and to facilitate discussion on how to enhance them.
Hot Tip: Role of Self-Assessment. In addition to external evaluation, it it important for program staff to engage in ongoing self-assessment processes which include POS program elements. External evaluators can help facilitate this process and provide constructive feedback/technical support throughout the process. When program staff have ownership of the evaluation and systemically review the quality of their after school program, enhancements to the program are more likely to be implemented.
Hot Tip: Evaluating Academic POS Elements. Often when assessing POS practices, either externally or through the self-assessment process, the focus of the assessment tends to be on youth development principles. Such principles might include interactions among youth and staff, safety, skill-building opportunities, social norms, and program routine or structure (Granger et al., 2007). Although such factors are important to students’ overall development, it is also important assess program elements that have been found to improve academic achievement, as identified in the IES practice guide (Beckett, et al., 2009). Elements identified by Beckett et al. that can be observed at the POS include:
- Aligning the out-of-school time program academically with the school day
- Maximizing student participation and attendance
- Adapting instruction to individual and small-group needs
- Providing engaging learning experiences
Therefore, evaluators should determine ways to assess these POS practices, both externally and through self-assessment.
By using these strategies, evaluators can help ensure out-of-school time evaluations comprehensively assess program quality. Instead of merely examining outcomes and providing program staff data after the program has ended, evaluators should focus on helping program staff identify POS elements that contribute to overall program quality. In this way, program staff will be able to assess current practices and implement changes prior to the assessment of program outcomes.
Want to learn more about evaluating out-of-school time programs? Consider attending skill building workshop 774 at Evaluation 2011, the American Evaluation Association’s Annual Conference this November. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.