AEA365 | A Tip-a-Day by and for Evaluators

Dec/13

20

MA PCMH Eval Week: Ann Lawthers, Sai Cherala, and Judy Steinberg on How You Define Success Influences Your Findings

Hi, we are Ann Lawthers, Sai Cherala, and Judy Steinberg, UMMS PCMHI Evaluation Team members from the University of Massachusetts Medical School’s Center for Health Policy and Research. Today’s blog title sounds obvious, doesn’t it? Your definition of success influences your findings. Today we talk about stakeholder perspectives on success and how evaluator decisions about what is “success” can change the results of your evaluation.

As part of the Massachusetts Patient-Centered Medical Home Initiative (PCMHI), the 45 participating practices submitted clinical data (numerators and denominators only) through a web portal. Measures included HEDIS® look-alikes such as diabetes outcomes and asthma care, as well as measures developed for this initiative, e.g., high risk members with a care plan. Policy makers were interested in whether the PCMH initiative resulted in improved clinical performance, although they also wanted to know “Who are the high- or low-performing practices on the clinical measures after 18 months in the initiative?” The latter question could be about either change or attainment. Practices were more interested in how their activities affected their clinical performance.

To address both perspectives we chose to measure clinical performance in terms of both change and attainment. We then used data from our patient survey, our staff survey, and the Medical Home Implementation Quotient (MHIQ) to find factors associated with both change and attainment.

Lesson Learned: Who are the high performers? “It depends.” High performance defined by high absolute levels of performance disproportionately rewarded practices that began the project with excellent performance. High performance defined by magnitude of change slighted practices that began at the top, as these practices had less room to change. The result? The top five performers defined by each metric were different.

Hot Tip:

  • Do you want to reward transformation? Choose metrics that measure change over the life of your project.
  • Do you want to reward performance? Choose metrics that assess attainment of a benchmark.
  • The results of each metric will include different lists of high performers.

Lesson Learned: The practices wanted to know: “What can we do to make ourselves high-performers?” Our mixed methods approach found leadership and comfort with Health Information Technology predicted attainment, but only low baseline performance predicted change.

Hot Tip: A mixed methods approach provides a rich backdrop for interpreting your findings and providing detail for stakeholders who need/want detail.

The American Evaluation Association is celebrating Massachusetts Patient-Centered Medical Home Initiative (PCMHI) week. The contributions all this week to aea365 come from members who work with the Massachusetts Patient-Centered Medical Home Initiative (PCMHI). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

2 comments

  • Ann Lawthers · January 17, 2014 at 12:37 pm

    The Evaluation Team provided results from the self-assessment MHIQ and other tools to the primary care practices and Project Team, from the beginning of the initiative. Thus, practices had continual feedback about their performance for quality improvement purposes. The Evaluation Team took periodic snapshots of performance and quantified high and low performance for evaluation purposes

    Reply

  • Tracey Stankus · December 21, 2013 at 6:25 pm

    I would think frequent assessment of the high performers from the evaluation team would help identify best practices. I am not sure if evaluators have built into the PCMHI a process which identifies low performers early in the process. Early detection of low performers will help identify practices that are contributing to the low performance. If evaluators use what they learned from following the high performers and there process they could provide guidance, education and training to low performers. Continued training and guidance to low performers should provide them with the tools they need to become high performers as long as they have the drive and incentive.

    Reply

Leave a Reply

<<

>>

Archives

To top