We are Michael Arnold, Senior Associate with Harder+Company Community Research where I design and implement social impact evaluations and Miranda Yates, Assistant Executive Director for Strategy, Evaluation, and Learning with Good Shepherd Services (GSS). This October at AEA 2016, we look forward to sharing some lessons from using Propensity Score Matching (PSM) in community program evaluations along with Michael Scuello and Donna Tapper of Metis Associates.
As more funders encourage or require grantees to evaluate program effectiveness, there is growing interest in alternatives to the Randomized Control Trial (RCT). RCTs are often impractical for programs that have eligibility criteria or participation drivers, which may also influence participant outcomes. For example, GSS Transfer Schools offer a full-day, year-round academic program that integrates intensive support services and youth development practices with personalized, standards-based instruction for students who have a history of truancy and are unlikely to graduate from high school before they turn 21. This eligibility criterion, while essential to the program design, complicates our efforts to measure program effectiveness unless we are able to locate an appropriate comparison group.
Comparing these students to the general population of students in the school districts is likely inappropriate because factors that distinguish eligibility (like falling behind academically) may also affect outcomes (like academic achievement). In this case, we could underestimate the true program effect. Ideally, we would want to match our enrolled youth to other similarly positioned youth who have not experienced GSS Transfer Schools. However, these other youth may be employed full-time or have other traits that might explain why they don’t participate in GSS Transfer Schools and influence their academic outcomes, which could lead to inflated program effect estimates. Without random assignment through RCT, it is possible to overstate or understate the true program effect.
PSM can be an effective approach for quasi-experimental studies where RCT is not feasible and where we wish to attribute outcomes to program participation. However, there are important considerations to take into account when deciding whether PSM is right for your evaluation, and identifying how to get the most from the approach. Here are some theoretical and practical introductions to using PSM for program evaluation.
Rad Resources:
- Wikipedia provides a good basic introduction to propensity score matching and definitions.
- There is also a helpful overview of PSM with tips and resources from Better Evaluation.
- If you prefer videos, Doug McKee presents “An Intuitive Introduction to Propensity Score Matching” (17 minutes).
The American Evaluation Association is celebrating Nonprofit and Foundations TIG Week with our colleagues in the NPF AEA Topical Interest Group. The contributions all this week to aea365 come from our NPFTIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.