I am Humberto Reynoso-Vallejo, Director of Research and Policy for the Office of the State Auditor in Massachusetts. This week the research team will share a series of useful tips for evaluating large and complex projects using our work evaluating the Health Care Cost Containment Law in Massachusetts known as Chapter 224.
Chapter 224 is a large and comprehensive health care initiative with the primary purpose of containing the growth of health care expenditures. The law also supports strategies for payment reform, for integrated health care delivery systems to increase access and quality of services for diverse population groups in different regions and for specific services such as primary care and behavioral health, promoting workplace wellness, for reducing the prevalence of preventable health conditions, and for reducing racial/ethnic disparities in health outcomes.
Conducting such a large and complex evaluation required the development of a comprehensive evaluation plan capturing all essential pieces of the law. The evaluation design needed to structure all the different topic areas into a logical and concise plan that guides all activities performed by a multidisciplinary team of researchers and data analytic experts. We proposed a longitudinal mixed-methods quasi-experimental design for the evaluation, aiming to determine the law’s impact on health care costs, access to health care services and quality of care, the health care workforce, and the impact on public health. Another important task to address in the evaluation plan is collecting and securing relevant data that will allow the team to perform both quantitative and qualitative analysis.
Hot Tips:
- Be specific when indicating the purpose of the evaluation. Include a brief but comprehensive overview and indicate who the target audience is.
- Divide the evaluation plan into manageable units, including high level evaluation questions that correspond to the overall topic areas, followed by principal evaluation questions on specific aspects of each topic area.
- After describing the purpose of the evaluation and stating the evaluation questions, write an introductory section that provides an overview of the project and any relevant information that allows the reader to understand more about the background of what you are evaluating.
- Develop a research design for the evaluation that includes methodology, data sources, sampling, and analysis strategy.
- Create a table with a set of measures for each principal evaluation question including their respective data sources. Also, indicate the analytical procedures for either quantitative or qualitative data.
- Create a timeline for evaluation activities, including research undertakings (collecting and analyzing data), engaging key stakeholders (as members of an Advisory Committee and/or research participants), creating interim reports, white papers, blogs, drafts, and other pertinent documentation for the evaluation process.
The American Evaluation Association is celebrating Large Scale Evaluation Week. The contributions all this week to aea365 come from evaluators who have worked on the evaluation of the Health Care Cost Containment Law in Massachusetts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.