I am David Keyes, Researcher, from The Oregon Community Foundation. The Research team at The Oregon Community Foundation (OCF) is currently conducting program evaluations of several OCF grantmaking initiatives, including an out-of-school-time initiative, intended to help close the achievement gap by supporting quality improvement and expansion of out-of-school time programming. Twenty-one grantee organizations with a variety of different program models are currently participating in the Initiative.
To learn about program quality and support program improvement, we are using the Youth Program Quality Assessment (YPQA) framework, tools, and training because it provides a low-stakes but research-based means of capturing program quality information that can be used to improve programming regardless of program model. The YPQA is a validated tool shown to be a predictor of programming quality. The assessment covers a range of topics and is completed using observation of programming, analysis of documents and interviews with staff.
Lessons Learned: The assessment process is rigorous. In the first year, initiative grantees completed online training provided by the Weikart Center before conducting an initial round of self-assessments. Grantees then received a detailed report, which they used to analyze their strengths and weaknesses and develop improvement plans. In the second year, the self-assessment process was repeated, and an external assessment was added. Several grantee staff members also received additional training and conducted assessments of other grantees.
The integration of the YPQA has also required intentional trust-building between the grantees, the evaluation team and the grant-making team to ensure that grantees engage in the YPQA process honestly and openly.
OCF has noted several benefits from using the YPQA in this initiative:
- Grantees now have a shared language about what quality programming looks like regardless of population-focus or program model;
- Grantee organizations are learning about research-based or promising practices for the work they are doing as well as how to implement related program improvements; and
- Grantee evaluation capacity is growing as they engage deeply in a relevant, systematic data collection and improvement process.
Hot Tip: Program quality assessment can be a valuable alternative to assessing fidelity to program models. The YPQA has enabled the research team at OCF to focus on spurring improvement in a context in which models of implementation for programming vary across grantees. Instead of assessing fidelity to a single model (which doesn’t exist in the initiative we are evaluating), we have focused on quality, no matter the programming model. This has allowed us to capture program information about programs in a way which serves our purposes of capturing information across our grantee group while also providing grantees data they can use to constantly improve their programming.
The American Evaluation Association is celebrating Oregon Community Foundation (OCF) week. The contributions all this week to aea365 come from OCF team members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.