BH TIG Week: Lessons learned in evaluating cross-systems programs by Lisa Melchior

Hi, I’m Lisa Melchior, President of The Measurement Group LLC, a consulting firm focused on the evaluation of health and social services for at-risk and vulnerable populations. I’d like to share some lessons I’ve learned in close to 30 years of working with community-based behavioral health and other programs that have addressed issues in populations with co-occurring substance use and mental health problems.

Effectively addressing multiple vulnerabilities in behavioral health is critical to optimizing participant outcomes. For example, Vivian Brown, George Huba, and I found that women in substance abuse treatment with high therapeutic burden (multiple co-occurring behavioral health, physical health, and other vulnerabilities) were more likely to terminate treatment early. However, if they were retained long enough, they were just as likely to succeed as other clients with less “burden.” Engaging and retaining participants long enough to benefit from the programs we evaluate is critical to achieving the intended outcomes of those interventions.

Hot Tip:

What does this mean for us as evaluators? Our evaluation designs, measures, and analyses need to address the multiple vulnerabilities that clients hope to address through the programs we evaluate – which are often addressed by multiple systems of care. The Transtheoretical Model can be a useful framework for approaching the measurement of cross-systems outcomes in client-focused evaluations. Yet we also need to be mindful of efficiency in our measurement and consider data collection within the context of the program.

Lessons Learned:

Don’t be limited by a program’s label or funding source. People in behavioral health programs have multiple needs, addressed by multiple systems. For example, employment may be an important predictor of subsequent success in reducing criminal justice system recidivism; as stated in a recent LA Times article, “in addition to substance abuse and mental health issues, chronic unemployment is one of the primary barriers to smooth re-entry.” If a program’s funding is for behavioral health services, don’t overlook including other indicators if they are pertinent to the intervention and its evaluation.

Take time to learn how multiple systems interact in the context of your program so the evaluation reflects those relationships accurately. For example, in a program we currently evaluate – a housing and treatment intervention for homeless young adults with behavioral health conditions – case-finding is conducted by a team specializing in outreach to the homeless, as opposed to staff from the behavioral health treatment team. As these are separate divisions within the organization, with different funders, it was important to understand these details and not make assumptions based on similar programs we previously evaluated.

As a practical issue, having a dedicated point person on the evaluation team who coordinates with program staff is critical! Especially with multisystem programs, there are many moving parts. Having an evaluation team member who is seen by program staff as an extension of their team is invaluable for ensuring high quality data.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.