BH TIG Week: A Call for a Systems Approach to Behavioral Health Evaluation by Roger A. Boothroyd

Hello, I’m Roger A. Boothroyd from the University of South Florida. One thing I have learned from conducting mental health services research over the past 30 years: Research repeatedly documents that approximately two-thirds of adults diagnosed with a mental health disorder have at least one physical health condition. It is also well known that comorbidity of mental health disorders and substance abuse disorders is high, ranging between 35 and 45%. Further, many adults with mental illness are likely to be arrested. Finally, over half of adults with a mental health disorder do not receive treatment. Thus: 1) mental health issues seldom occur by themselves but often occur with other comorbid conditions; and 2) of adults with mental health disorders who enter treatment, comorbid conditions often result in them being served simultaneously by multiple service systems.

For children and youth with emotional and behavioral challenges, the issue of simultaneous multiple service system involvement is even more complex. Children and youth attend school, so the educational system is necessarily involved. Often, they are involved with the child welfare and/or juvenile justice systems; and, of course, their families play a significant role in their day-to-day lives. Thus, the question for us as evaluators is: How can we realistically evaluate the effectiveness of a program or an intervention without assuming a more systems level evaluative perspective?

Lesson Learned: Some 20 years ago, I was involved in an evaluation that explored why so few adults with severe mental illness who sought vocational rehabilitation services received them and were successful in obtaining jobs. Our evaluation included a systems thinking framework that involved modeling how individuals with severe mental illness entered and moved through the mental health and vocational rehabilitation systems. At the start of the evaluation, the prevailing hypothesis (mine included) was that there were not enough resources available for vocational rehabilitation services for adults with severe mental illness. Yet, when the cross-systems model was constructed, many adults with severe mental illness were receiving vocation rehabilitation services. The real problem was the lack of sufficient numbers of jobs for those adults who were trained; and the lack of jobs prevented them from exiting the vocational rehabilitation system. In fact, the model predicated that if more resources had been devoted to vocational rehabilitation services, the functioning of both systems would have gotten much worse. The answer was straightforward: Open up more jobs. The county mental health and vocational rehabilitation departments worked together with their Chamber of Commerce and local businesses to secure job placements for adults who had completed vocational rehabilitation training. As the flow of adults through these systems improved, the capacity to train other adults increased – all without new resources. This was my first introduction into systems thinking and seeing firsthand the importance of assuming a broader evaluation perspective.

This week, evaluators from the Behavioral Health (formerly Alcohol, Drug Abuse, and Mental Health) Topical Interest Group will share their strategies, experiences, and insights gained from conducting behavioral health-related evaluations that assumed this broader systems-level perspective.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.