Linda Cabral and Laura Sefton on So many to choose from: How to Select Organizations for a Site Visit

Greetings. We are Linda Cabral and Laura Sefton from the University of Massachusetts Medical School, Center for Health Policy and Research. We are part of a multi-disciplinary team evaluating the Massachusetts Patient Centered Medical Home Initiative (MA PCMHI), a 3-year, statewide, multi-site demonstration project engaging 46 primary care practices in organizational transformation to adopt this primary care model of care.  Part of the qualitative component to our evaluation included site visits to a select group of practices to interview project-related staff. Given the project’s resources, the team was only able to visit 8 sites. So with 46 practices to choose from, how did we go about selecting which 8 to visit?

In order to answer our primary evaluation question – how do practices transform to become medical homes? – we knew that we wanted to visit practices that had transformed substantially, as well as those that had not (to date).  Each practice had staff complete the TransforMED’s Medical Home Implementation Quotient (MHIQ) survey at baseline and midpoint. Each practice, therefore, had data at two time points that could be a measure of how much they have adopted medical home competencies.

To identify practices with a ‘high’ or ‘low’ change rate, we assigned each practice an MHIQ change scale ranking.  We calculated a Midpoint-Baseline change score and then ordered the change scores from lowest to highest. We assigned a quintile rank to each practice as follows:

1= Lowest change rate (80% of scores showed more change)

2= Low change rate (60% of scores showed more change)

3= Medium change rate (40% of scores showed more change)

4= Better change rate (20% of scores showed more change)

5= Highest change rate (Showed most change)

Alongside each practice’s MHIQ ranking, we displayed other sources of data, including a self-reported Practice Transformation Status measure and several clinical quality scores on standard measures such as diabetes control, depression screening rates, and rates of follow-up after a hospital discharge.

With these data available to us, we next consulted with the MA PCMHI Quality and Transformation Director to verify the data’s validity and assist in selecting practices for recruitment.  Subsequently, five ‘high change rate’ practices and three ‘low change rate’ practices were identified.

Hot Tips:

• Use multiple sources of data, including qualitative data, to inform site selection. This allows for a broader picture of each site, allowing us to make selections based on a range of variables.
• Identify a set of alternate sites. No matter how good your site selection process, any site can opt out of participating.

Rad Resources: This is our favorite comprehensive toolkit for qualitative research. One of the volumes includes a guide to site visit data collection.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.