Hi, we are Humberto Reynoso-Vallejo and Qiong Louie-Gao, members of the research team evaluating the Health Care Cost Containment Law (Chapter 224 of the Acts of 2012) from the Office of the State Auditor in Massachusetts.
A strategy that has been used successfully to investigate the effect of policy changes over time on population outcomes is Interrupted Time-Series (e.g. Ramsey, et al., 2003). Since our project requires us to collect multiple data points previous to the implementation of the Chapter 224 law, as well as several points after, Interrupted Time-Series, for aggregate data, will allow us to determine whether the health care costs containment intervention has an effect significantly greater than a secular trend. By using this method, we will be primarily testing the change in the slope of data trends as a function of Chapter 224.
Hot Tips:
- Data Collection: As a rule of thumb, 10 measurements pre-event and 10 measurements post-event are suggested for adequate power to detect change. Considering that having that number of pre and post event measurements is not always attainable, try to collect as many data points as possible previous to and after the policy implementation.
- Analysis: Co-occurring events that may influence outcomes should be considered, as well as any changes in measurement or reporting methods for variables of interest.
- Lagged Effect: Because the effect of implementation of policy changes can be delayed beyond the start date of a law, the evaluation report should provide the foundation for any future analysis of impact beyond the evaluation project end date.
- Autocorrelation: Time points collected in a relatively close time frame may be correlated with each other. Ignoring autocorrelation leads to underestimation of the standard errors and overestimation of the statistical significance.
- For time-series analysis, we are planning to use segmented regression analysis (Wagner et al., 2002). This approach will allow us to estimate and test the trend pre and post the implementation of Chapter 224, and to determine the change in level when the intervention is introduced.
?t = ?0 + ?1 x timet + ?2 x interventiont + ?3 x time after_interventiont + et
Adapted from: Ramsey, et al., 2003.
The American Evaluation Association is celebrating Large Scale Evaluation Week. The contributions all this week to aea365 come from evaluators who have worked on the evaluation of the Health Care Cost Containment Law in Massachusetts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.