Hello! We’re Allan Porowski from ICF International and Heather Clawson from Communities In Schools (CIS). We completed a five-year, comprehensive, mixed-method evaluation of CIS, which featured a several study components – including three student-level randomized controlled trials; a school-level quasi-experimental study; eight case studies; a natural variation study to identify what factors distinguished the most successful CIS sites from others; and a benchmarking study to identify what lessons CIS could draw from other youth-serving organizations. We learned a lot over the years, and wanted to share a few big takeaways with you about conducting evaluations on interventions for at-risk youth.
Lessons Learned:
- Sometimes, you have to catch falling knives: We found that the students coming into CIS were targeted for services because they were on the strongest downward trajectories on a number of factors (e.g., academics, behavior, family issues, attendance, etc.). There’s an old adage in stock market trading that you should “never catch a falling knife” – but that’s what CIS and other dropout prevention programs do every day. This has implications for how you evaluate the relationship between dosage and outcomes. A negative relationship between dosage and outcomes doesn’t necessarily indicate that services aren’t working – it can actually be an indication that services are going to where they are needed the most.
- Look for the “Nike Swoosh”: The general pattern of outcomes among CIS students looked like Nike’s “swoosh” logo: There was an initial downward slide followed by a longer, more protracted period of improvement. Reversing that initial downward slide takes time, and this pattern is worth investigating if you’re evaluating programs for at-risk youth.
- As the prescient rock band Guns n’ Roses put it, “All we need is just a little patience”: Needless to say, it takes a long time to turn a child’s life around. So many evaluations of at-risk students don’t have a long enough time horizon to show improvements, which may in part explain why we see such low effect sizes in dropout prevention research relative to other fields of study.
Rad Resources:
- Executive Summaryof Communities In School’s Five-year National Evaluation
- Communities In Schools has great ideas and resources for dealing with at-risk youth. CIS surrounds students with a community of support, empowering them to stay in school and achieve in life. Through a school-based coordinator, CIS connects students and their families to critical community resources, tailored to local needs. Working in nearly 2,700 schools, in the most challenged communities in 25 states and the District of Columbia, CIS serves nearly 1.26 million young people and their families every year.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.