AEA365 | A Tip-a-Day by and for Evaluators

CAT | Behavioral Health

I am Faisal Islam, Evaluation Specialist at Ontario Shores Centre for Mental Health Sciences in Whitby, Canada. Recently, I conducted an evaluation of a mental health court in Ethiopia. First of its kind in Ethiopia, the mental health court was established in collaboration with a number of partners from Ethiopia and Canada. As you can imagine, the evaluation not only dealt with systems and cultures in the two countries, but also with a variety of stakeholders: judges, psychiatrists, correction centre’s staff, police, social workers, and of course, persons with mental health and legal challenges and/or their families.

Hot Tips:

Active awareness of self and others can help the evaluator work for a process that has a fair representation of multiple systems and diverse stakeholders. Overlooking some stakeholders and overly recognizing others can jeopardize the good work that evaluation can bring. Remember: An evaluator is steering the process of evaluation. S/he must be cognizant of power dynamics, organizational cultures and competing interests and use her/his position to assure inclusion.

Openness and willingness to reach-out, negotiate, surrender and adapt can help the evaluator to work for an evaluation framework that has a buy-in. Start early and create appropriate spaces to obtain feedback on the evaluation process. Phone calls, emails, and physical interactions with follow ups can matter in creating shared understanding, building trust and seeking collaborations/ engagements.

Understand inter-relationships and be flexible and respectful in methods of engagement and data collection. Judges, psychiatrists, police, families, and social workers all represent different systems and thus have different priorities and functioning in the mental health court. Yet, the court connects them to perform.  Use system approaches to understand inter-relationships between systems and build evaluation around those relationships.

 Rad Resources:

I found the 2004 special issue of New Directions in Evaluation In Search of Cultural Competence in Evaluation: Toward Principles and Practices extremely helpful in conceptualizing participation and inclusiveness of cross-systems in evaluation.

Bob Williams’ and Iraj Imam’s System Concept in Evaluation: An Expert Anthology is a good resource to understand how systems thinking can be applied in evaluations.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi! I’m Shannon Campbell, senior research/evaluation analyst for Multnomah County’s Mental Health and Addiction Services Division (MHASD). Beyond its responsibilities as the local mental health authority, MHASD also partners with Health Share of Oregon to co-manage Medicaid behavioral health benefits.

Recognizing the links between physical and behavioral health, reducing emergency department (ED) visits, is an important outcome for many of our programs. I recently analyzed how substance use and mental health intersect with ED visits for our Medicaid population—a dataset of 270,000 visits over 3.5 years. This first analysis was intended as a foundation for future work—to get the ‘big picture’ of what was happening in our hospitals, and later connect that data to our clients and services.

The first, most simple approach would be to calculate how many visits claimed behavioral health diagnoses. However, this approach may be insufficient if our goal is to understand not only the obvious links between behavioral and physical health (e.g., overdoses or psychoses), but also the more subtle—for example, if someone is admitted for something that could be addiction-connected, but doctors do not take (or have) the time for standard substance use screenings, or a homeless patient presents with minor complaints in hopes of a warm bed, but their housing status is closely connected with their mental health.

Thus, approach #2: Calculate how many patients had behavioral health diagnoses in at least one ED visit, and flag those patients’ total visits that same year (e.g., if “Bob” had one 2015 visit noting alcoholism, flag all of his 2015 visits).

This expanded the picture further, but was it enough? I attempted one more approach: Calculate how many patients had behavioral health diagnoses in any claim (primary care, mental health services, etc.), and flag all of their ED visits [within the same year(s) the diagnosis was recorded].

Yet, when is a broken arm merely a broken arm? Is this overreach? Are we back to square one—stick with what the claim said?

The answer: Probably somewhere in-between. Further investigation is required!

Lesson Learned:

Cherry-picking methods to tell certain stories rightly deserves criticism. However, different results under different approaches can be a helpful outcome in and of itself and offers interesting new avenues of inquiry; e.g., looking at success in adopting behavioral health screening tools, the presence of specific social determinants, or what physical ailments may be signs of other underlying issues. The methods became part of the results by the questions they raised. To riff on Marshall McLuhan, sometimes the method is the message.

Hot Tip:

Find your tribe. Many of us work independently. Find people with whom you can geek out about methods, data, and the other questions you should be asking without worrying about the actual evaluation results. Bringing this project to a regional data workgroup of supportive colleagues provided great feedback, which encouraged me to reach out to my contacts more frequently to informally ‘shoot the breeze’ about current projects. Tap into your network!

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Greetings from Saumitra SenGupta and Gale Berkowitz.  Together we lead the External Quality Review of 56 California Medi-Cal managed specialty mental health plans (MHPs) through Behavioral Health Concepts.

Each year we are required to review all 56 MHPs, produce a report for each, AND produce a statewide annual report. Our role is more to validate the extent to which the plans adhere to standards around access, timeliness, quality and outcomes, from a quality improvement perspective (versus traditional evaluation of these systems).  We have to do this quickly, consistently, and with reliability and validity as we adhere to protocols from the Centers for Medicare and Medicaid.

California is large and complex.  If California were a nation, it would be the seventh largest.  Along with its sheer size comes diversity and complexity, and this is reflected not only in the mental health systems but among the consumers in those systems. No MHPs are alike in terms of system organization, population served, size, and other characteristics.

So how do we balance the demands of adhering to required protocols while at the time articulating the unique aspects of each MHP’s strengths, challenges and opportunities?

Lessons Learned:

Takeaway #1: To obtain a 360-degree view of complex systems, use multiple sources of information, employ multi-platform, mixed-method designs.

We rely on triangulation across multiple sources of data gathered from pre-site planning, on-site review, and report production to gather a 360-degree view of the system. These include co-development of the review agenda with the MHPs; analysis of paid claims, document review, site visits, and on-site interviews with leadership, clinical staff, information systems staff, and beneficiaries receiving services. Before a report is finalized, the MHP is asked to review and give feedback on the report.

Takeaway #2: For cross-system, large scale reviews and evaluations, pay special attention to developing relevant protocols, questionnaires, and templates.

For all phases of the reviews, we have developed protocols, tools, and templates that we follow, such as session guidelines, standard performance measures, interview guides, rating systems, checklists, and a report template. While these are more guidelines than rules, they ensure that our teams follow generally the same protocols and ask the same questions to be able to complete necessary analyses.

Takeaway #3:  Build ongoing opportunities for peer review and training.  As we mentioned, we do not do this alone.  We are supported by a diverse team that come from varied professional and educational backgrounds themselves. To build a consistent voice and ensure that we build reliability within and across reviews, we conduct regular trainings, include everyone in the development and revision of all protocols, and rely on peer assists and peer review of each other’s reports.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Evangeline Danseco and I work as a Performance Measurement Coach at the Ontario Centre of Excellence for Child and Youth Mental Health. I help community-based non-profit agencies develop a performance measurement framework for their organization.

In my recent work with community-based agencies, I encourage them to think about including both SMART indicators and WISE indicators in their performance measurement framework. Key performance indicators let an organization know how well it is progressing toward achieving the goals they set out. Indicators that are SMART are those that are Specific, Measurable, Achievable, Relevant and Timely. When indicators are framed the SMART way, there is greater clarity among staff when collecting and analyzing the data.

Sometimes, it is difficult to summarize our work with numbers using the SMART criteria. The stories and inspiring elements of the work do not surface enough and yet exert a tremendous influence on the organization’s development and culture. WISE indicators consider the Whole system, are Inspiring, consider the Story and the Synergy among indicators, and are Engaging. Considering the Whole system means that a holistic perspective of the organization’s mission is evident among the indicators, and that a systems approach or a theory of change is reflected among all of the indicators. Indicators can be inspiring and point to the things that the organization is improving upon, not only focus on errors or mandatory requirements. The Story and Synergy among the indicators provide an accurate picture of what is happening in the entire organization. Finally, Engaging indicators need to involve key stakeholders at every step of the process of defining, collecting and using the indicators. When identifying key performance indicators, make sure that the indicators can galvanize people into action. Try to have space for these WISE indicators that are important and complement other indicators that are measured in traditional ways.

Hot Tip: A scorecard is a useful way of summarizing key performance indicators that agencies want to monitor and improve upon. Domains in a scorecard typically include financial indicators, staff or human resource indicators, program effectiveness or impact, and indicators relating to the key stakeholders of the agency.

Rad Resource: Our recent report summarizes some of the domains and examples of child and youth mental health indicators measured at a system level such as state-wide or country level indicators. These indicators help different stakeholders such as policy-makers, researchers, families and clinicians understand how well a system is doing.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, I’m Lisa Melchior, President of The Measurement Group LLC, a consulting firm focused on the evaluation of health and social services for at-risk and vulnerable populations. I’d like to share some lessons I’ve learned in close to 30 years of working with community-based behavioral health and other programs that have addressed issues in populations with co-occurring substance use and mental health problems.

Effectively addressing multiple vulnerabilities in behavioral health is critical to optimizing participant outcomes. For example, Vivian Brown, George Huba, and I found that women in substance abuse treatment with high therapeutic burden (multiple co-occurring behavioral health, physical health, and other vulnerabilities) were more likely to terminate treatment early. However, if they were retained long enough, they were just as likely to succeed as other clients with less “burden.” Engaging and retaining participants long enough to benefit from the programs we evaluate is critical to achieving the intended outcomes of those interventions.

Hot Tip:

What does this mean for us as evaluators? Our evaluation designs, measures, and analyses need to address the multiple vulnerabilities that clients hope to address through the programs we evaluate – which are often addressed by multiple systems of care. The Transtheoretical Model can be a useful framework for approaching the measurement of cross-systems outcomes in client-focused evaluations. Yet we also need to be mindful of efficiency in our measurement and consider data collection within the context of the program.

Lessons Learned:

Don’t be limited by a program’s label or funding source. People in behavioral health programs have multiple needs, addressed by multiple systems. For example, employment may be an important predictor of subsequent success in reducing criminal justice system recidivism; as stated in a recent LA Times article, “in addition to substance abuse and mental health issues, chronic unemployment is one of the primary barriers to smooth re-entry.” If a program’s funding is for behavioral health services, don’t overlook including other indicators if they are pertinent to the intervention and its evaluation.

Take time to learn how multiple systems interact in the context of your program so the evaluation reflects those relationships accurately. For example, in a program we currently evaluate – a housing and treatment intervention for homeless young adults with behavioral health conditions – case-finding is conducted by a team specializing in outreach to the homeless, as opposed to staff from the behavioral health treatment team. As these are separate divisions within the organization, with different funders, it was important to understand these details and not make assumptions based on similar programs we previously evaluated.

As a practical issue, having a dedicated point person on the evaluation team who coordinates with program staff is critical! Especially with multisystem programs, there are many moving parts. Having an evaluation team member who is seen by program staff as an extension of their team is invaluable for ensuring high quality data.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, I’m Roger A. Boothroyd from the University of South Florida. One thing I have learned from conducting mental health services research over the past 30 years: Research repeatedly documents that approximately two-thirds of adults diagnosed with a mental health disorder have at least one physical health condition. It is also well known that comorbidity of mental health disorders and substance abuse disorders is high, ranging between 35 and 45%. Further, many adults with mental illness are likely to be arrested. Finally, over half of adults with a mental health disorder do not receive treatment. Thus: 1) mental health issues seldom occur by themselves but often occur with other comorbid conditions; and 2) of adults with mental health disorders who enter treatment, comorbid conditions often result in them being served simultaneously by multiple service systems.

For children and youth with emotional and behavioral challenges, the issue of simultaneous multiple service system involvement is even more complex. Children and youth attend school, so the educational system is necessarily involved. Often, they are involved with the child welfare and/or juvenile justice systems; and, of course, their families play a significant role in their day-to-day lives. Thus, the question for us as evaluators is: How can we realistically evaluate the effectiveness of a program or an intervention without assuming a more systems level evaluative perspective?

Lesson Learned: Some 20 years ago, I was involved in an evaluation that explored why so few adults with severe mental illness who sought vocational rehabilitation services received them and were successful in obtaining jobs. Our evaluation included a systems thinking framework that involved modeling how individuals with severe mental illness entered and moved through the mental health and vocational rehabilitation systems. At the start of the evaluation, the prevailing hypothesis (mine included) was that there were not enough resources available for vocational rehabilitation services for adults with severe mental illness. Yet, when the cross-systems model was constructed, many adults with severe mental illness were receiving vocation rehabilitation services. The real problem was the lack of sufficient numbers of jobs for those adults who were trained; and the lack of jobs prevented them from exiting the vocational rehabilitation system. In fact, the model predicated that if more resources had been devoted to vocational rehabilitation services, the functioning of both systems would have gotten much worse. The answer was straightforward: Open up more jobs. The county mental health and vocational rehabilitation departments worked together with their Chamber of Commerce and local businesses to secure job placements for adults who had completed vocational rehabilitation training. As the flow of adults through these systems improved, the capacity to train other adults increased – all without new resources. This was my first introduction into systems thinking and seeing firsthand the importance of assuming a broader evaluation perspective.

This week, evaluators from the Behavioral Health (formerly Alcohol, Drug Abuse, and Mental Health) Topical Interest Group will share their strategies, experiences, and insights gained from conducting behavioral health-related evaluations that assumed this broader systems-level perspective.

The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Ann Price and I am the President of Community Evaluation Solutions, Inc. (CES), a consulting firm based in Alpharetta, Georgia just outside of Atlanta. I am a community psychologist and work with many federally-funded, community-based alcohol and substance abuse prevention programs. Across the state of Georgia, whether in rural areas or suburban areas of the state, community coalitions are working together to prevent youth substance use using environmental strategies.

Several of the Georgia prevention collaboratives with whom I work are using Positive Social Norms (PSN) campaigns to prevent youth substance abuse. Many choose either the “Most of Us” approach taught by the Montana Institute or the methodology proposed by Michael Haines. Both involve developing substance abuse prevention messages to correct incorrect perceived norms of rates of use. The goal of the PSN approach is to bring perceived norms in line with actual norms of alcohol or other substance use. The hope is that by correcting the perceived norm, the rates of underage drinking and binge drinking will decrease. For example, many youth overestimate actual rates of alcohol use on college campuses, assuming that ALL students drink, when in fact, this is not true. A PSN campaign might include messages like “Most Central High School students don’t drink” that are shared on campus posters and through social media. Some of our clients are beginning to demonstrate a correction in the perceived norm and a reduction in youth alcohol use.

Lessons Learned: Community-based programs are most effective when they are grounded in the needs of the community and reflect the “But why here?” That is, the factors in the community that support teen initiation and use and misuse of alcohol or other drugs.

Hot Tip: Think of the “But Why?” as the overarching cause of a social problem such as substance abuse and the “But Why here?” as the local condition helps communities focus on what is really driving a particular issue in their community. Designing the logic model, community intervention, and the evaluation around the “But Why?” and “But Why Here?” helps to focus your work.

Lesson Learned: You can lead a horse to water but you can’t make your clients drink. Coalitions and community collaboratives get stuck along the prevention road in many different ways. For example, some get stuck in the beginning phases of coalition development. Others get stuck after they design their logic model but never move on to implementation. Some talk about, but never develop a sustainability plan. Evaluation is also about program development and implementation- be there for your client as a prevention partner every step of the way.

Rad Resource: Community Anti-Drug Coalitions of America (CADCA) is a great resource for prevention information.

We’re looking forward to October and the Evaluation 2016 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

My name is Gisele Tchamba and I work for a faith-based organization where I lead in evaluation capacity building. I am also one of the ADAMH TIG leaders.  Today I want to share my experience with the implementation of a new treatment model for behavioral health providers in my county.

The Recovery Oriented Systems of Care Model (ROSC) model is a network of formal and informal services developed and mobilized to sustain long-term recovery for individuals and families impacted by severe substance use and mental health disorders. This model was created to replace the Acute Care model that was not very successful at providing long-term recovery outcomes.

Lesson Learned: The ROSC model is the dream treatment for behavioral health field. However, its implementation has been in process for over 10 years without success. I compared the implementation of the recovery-oriented systems of care model to the Transtheoretical model of behavior change and evaluated my county’s efforts to create its own recovery oriented systems of care model.  This model posits that health behavior change involves progress through six stages of change: precontemplation, contemplation, preparation, action, maintenance, and termination. I used this model because of the complex process in the county’s attempt to implement the process. The evaluation revealed that the county was in the second stage of the TTM model (the contemplation stage). But today my county is in the Action stage of the implementation and ready for another evaluation. In reality the ROSC is not a myth, it’s just very difficult to implement.

Hot Tips

  • Each state can create its own ROSC model by forming a task force that can comprise four workgroups (funding, prevention, screening and assessment, and treatment and recovery). These workgroups represent segments of the recovery oriented systems of care model.
  • A successful way to implement the ROSC model in a given community requires commitment from each provider.
  • To facilitate dialogue between providers, there must be funding for liaison. That element is currently missing in my county. I think that must be the reason for the slow progress noted so far.

Lesson Learned: A successful implementation of the recovery oriented systems of care model in this county will lead to the following:

  • Best practice, increase knowledge and facilitate evaluation
  • The county will become a recovery community with sustained long-term recovery outcomes for individuals and families; and
  • There will be better allocation of resources, better community involvement, and honest dialogue among providers.

Rad Resources

The Role of Recovery Support Services in Recovery-Oriented Systems of Care White Paper.

Willam White, an experienced researcher, offers a definition of the ROSC Model.

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings AEA365 community.  My name is Paul St. Roseman, and I am the owner/principal investigator with DataUse Consulting Group.  The intersection of evaluation and professional development has been an interest of mine throughout my 20-year career as an educator and evaluation researcher.  Within my practice, I have observed how participation in evaluations impact how people: (a) view themselves professionally, (b) approach their work, (c) communicate with one another, and (d) discuss the operation of their organization and its impact within the community.

Recently, I have begun to consider what these influences suggest about an evaluation’s overall impact on both learning and leadership development within organizations with whom I work.  To this aim, I have developed an Evaluation Capacity Building Learning and Leadership Conceptual Framework that I use to monitor how learning and leadership development within an organization transition as a collaborative evaluation study is completed.

Connecting evaluation within learning and leadership development allows me to examine the impact of an evaluation study beyond documenting organizational processes and accounting for services.  With the addition of these lenses, it is possible to locate the professional and organizational development that accompanies collaborative evaluation efforts in ways that are personal, contextual, transparent and immediate.  While the indicators used in this framework are far from perfect, they do provide a glimpse into what is possible when multiple theoretical lenses are used to push understanding regarding the experiences and lessons that accompany an evaluation experience.

Lesson Learned: Evaluation’s influence on client development is strengthened when efforts are professionally relevant and provide increased opportunities for participation across an organization.

Lesson Learned: The benefits of evaluation efforts deepen when clients are motivated, emotionally connected, and trusting of the experience.

Lesson Learned: In instances where reflective dialogue occurs between clients the potential for mutual learning increases and leadership capacities shift.

Lesson Learned: When clients develop the capacity to leverage existing and new skills within an evaluation study, confidence and self-determination applying evaluation findings to decision making strengthen.

Lesson Learned: Newly developed evaluation processes and skills can be difficult to sustain and integrate within an organization.  This is the case even when such efforts are valued by stakeholders.  Given this factor, the benefits of evaluation capacity building may be most apparent in multi-year evaluations that incorporate capacity building workshops, coaching, and technical support alongside data collection and analysis efforts.

Roseman

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Julie Slay, Senior Director of Evaluation at Arabella Advisors, a consulting firm that helps philanthropists and investors achieve the greatest good with their resources. Often, we are asked to evaluate a funder’s portfolio or initiative using indicators based on secondary data sets that reflect community-level and individual-level outcomes. Administrative data have potential to help us do that for behavioral health – or does it?

In the world of health care, using administrative claims data is attractive for a number of reasons. First, what are claims data? They are large data sets, held by public and private payors, that contain information about what a health care provider submitted to get paid for their service. Recent claims data, thanks to HIPAA, are in digital form – downloadable data spreadsheets that provide information about each client health encounter. But how accurate are those claims data? Do they really tell you the story of behavioral health care? Probably not.

Lesson Learned: Claims data have a number of constraints that are important to remember.

  1. Claims data reflect what the provider and payor need, not always what happened. Studies continue to compare claims data with service encounter data and find significant discrepancies. Often, claims data underestimate the amount and type of services provided, because providers know their payors’ rules and will not submit claims for a non-reimbursable services.
  2. Data required for reimbursement may not provide you with an accurate picture of quality. Mental health conditions are chronic; over one’s lifetime, there may be shorter, intense periods of using services, and longer periods where fewer services are used. However, we cannot know just from patterns of service use whether the client is receiving high quality services. High quality services might be reflected in a greater frequency of seeing a provider, but it also may indicate poor quality that requires multiple visits to get the care that one needs.
  3. Claims data are often dated. Once a service is provided, the data must be cleaned before it is claimed. Once sent to a payor, it might go back and forth between the payor and provider due to an error, or a denial then an appeal, and after several months, the service is paid for, or not. So in a world where people want real-time data, administrative claims have NO potential for that.

Lesson Learned: Try using clinical data from an electronic health record (EHR) instead of claims data to assess outcomes. Rules related to reimbursement should not prevent providers from entering these service, and data are close to real-time. If you use claims data, do not rely on it as the only measure of an outcome; use other indicators from different data sources to augment your findings.

The American Evaluation Association is celebrating Alcohol Drug Abuse and Mental Health (ADAMH) TIG Week. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top