Hi, we are Cathy Lesesne, Lorin Boyce, Lori Zakalik, Michele D Sadler, Erin Malone, and Drew Derrico from the Evaluation and Research for Action Center of Excellence at Deloitte Consulting LLP. We are excited to share some tips for using extant data to conduct or supplement impact evaluations of health care programs and services.
The proliferation of secondary data sources is changing our evaluation field. The data available from electronic health records, health care payors, and data lakes linking sources from across previously siloed repositories all have increasing use in evaluation. As evaluators we need to consider various approaches to sourcing evaluations. In some cases, secondary data sources that may offer better insight than what we could gather prospectively with limited resources and time. With these realities in mind, critical questions an evaluator should ask in designing a health-focused evaluation are: How necessary is it to collect primary data for this evaluation? Are there alternative, secondary sources? What are the costs and benefits of using secondary data sources instead of or in addition to primary data sources?
Secondary sources are often limited in the outcomes available in these data which can make them a compromise position. However, we can use these data in quasi-experimental designs such as regression discontinuity, difference in difference, matched comparison groups, and/or forecasting/modeling to help us answer high-level outcome and impact evaluation questions like never before. Coupled with qualitative primary data collection, evaluations can be quite rigorous and completed rapidly.
For a recent evaluation we are leading, we found the use of EPIC patient pool data was both the best outcome data source for an evaluation of a healthy lifestyle intervention delivered by a health system and it was also the best option to create a well-matched comparison group. This design choice ensured that our counterfactual arm was robust while also managing the need to provide this program to anyone eligible to participate. Our use of secondary health care data allows us to rapidly look at health outcomes and examine cost effectiveness, all while having a strong matched-comparison patient population. Additional qualitative elements were added to our evaluation to provide more context on the experience of participants, program delivery, and implementation.
Hot Tip
The following are examples of data repositories that we have found broadly useful and have data at various levels of specificity and geography. This list is not exhaustive, we have no relationship or conflict with any of these repositories, these represent our professional opinions alone, and do not represent the opinion of Deloitte Consulting LLP. They are simply helpful leads.
- Research Data Assistance Center (ResDAC) is a Centers for Medicare and Medicaid Services (CMS) resource for exploring, accessing, and using CMS data for a wide range of research and forecasting activities. Find, Request and Use CMS Data | ResDAC
- Healthcare Cost and Utilization Project’s (HCUP) Nationwide databases can be used to identify, track, and analyze national trends in healthcare utilization, access, charges, quality, and outcomes. HCUP’s State-specific databases can be used to investigate State-specific trends in healthcare utilization, access, charges, quality, and outcomes. HCUP-US Databases (ahrq.gov)
- Epic Cosmos is a dataset created in collaboration with a community of health systems using Epic and is designed to improve patient care. Working with health programs using EPIC health record system? Ask if they are connected to EPIC Cosmos. Epic Cosmos
- Inter-university Consortium for Political and Social Research (ICPSR) is part of the Institute for Social Research at the University of Michigan. ICPSR provides research science data and resources on topics like social media, politics, economics, social sciences, government, GIS, and more. Find Data (umich.edu)
The American Evaluation Association is hosting the Cluster, Multi-Site, and Multi-Level Evaluation (CMME) TIG Week. The CMME TIG is encompasses methodologies and tools for designs that address single interventions implemented at multiple sites, multiple interventions implemented at different sites with shared goals, and the qualitative and statistical treatments of data for these designs, including meta-analyses, statistical treatment of nested data, and data reduction of qualitative data. The contributions all this week to AEA365 come from our CMME TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.