Greetings from Saumitra SenGupta and Gale Berkowitz. Together we lead the External Quality Review of 56 California Medi-Cal managed specialty mental health plans (MHPs) through Behavioral Health Concepts.
Each year we are required to review all 56 MHPs, produce a report for each, AND produce a statewide annual report. Our role is more to validate the extent to which the plans adhere to standards around access, timeliness, quality and outcomes, from a quality improvement perspective (versus traditional evaluation of these systems). We have to do this quickly, consistently, and with reliability and validity as we adhere to protocols from the Centers for Medicare and Medicaid.
California is large and complex. If California were a nation, it would be the seventh largest. Along with its sheer size comes diversity and complexity, and this is reflected not only in the mental health systems but among the consumers in those systems. No MHPs are alike in terms of system organization, population served, size, and other characteristics.
So how do we balance the demands of adhering to required protocols while at the time articulating the unique aspects of each MHP’s strengths, challenges and opportunities?
Lessons Learned:
Takeaway #1: To obtain a 360-degree view of complex systems, use multiple sources of information, employ multi-platform, mixed-method designs.
We rely on triangulation across multiple sources of data gathered from pre-site planning, on-site review, and report production to gather a 360-degree view of the system. These include co-development of the review agenda with the MHPs; analysis of paid claims, document review, site visits, and on-site interviews with leadership, clinical staff, information systems staff, and beneficiaries receiving services. Before a report is finalized, the MHP is asked to review and give feedback on the report.
Takeaway #2: For cross-system, large scale reviews and evaluations, pay special attention to developing relevant protocols, questionnaires, and templates.
For all phases of the reviews, we have developed protocols, tools, and templates that we follow, such as session guidelines, standard performance measures, interview guides, rating systems, checklists, and a report template. While these are more guidelines than rules, they ensure that our teams follow generally the same protocols and ask the same questions to be able to complete necessary analyses.
Takeaway #3: Build ongoing opportunities for peer review and training. As we mentioned, we do not do this alone. We are supported by a diverse team that come from varied professional and educational backgrounds themselves. To build a consistent voice and ensure that we build reliability within and across reviews, we conduct regular trainings, include everyone in the development and revision of all protocols, and rely on peer assists and peer review of each other’s reports.
The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.