AEA365 | A Tip-a-Day by and for Evaluators



Linda Cabral and Jillian Richard-Daniels on Using Qualitative Data Collection to Better Understand Measure Development and Reporting

Hello, we are Linda Cabral and Jillian Richard-Daniels from the Center for Health Policy and Research at University of Massachusetts Medical School. Collecting and reporting data on a set of predefined measures is something that many evaluators are asked to do.  This typically quantitative process involves gathering data, often from multiple sources and stakeholders, to assess the amount, cost, or result of a particular activity. But have you ever thought about what goes into measure development, collection and reporting? A recent evaluation that we completed included interviews with the people involved in this process.

A federal grant interested in testing a set of child health quality measures was awarded to a group of stakeholders in Massachusetts. An example of such a quality measure is the percentage of infants who reached age 15 months during the measurement year, and who has six or more well infant visits during the first 15 months of life. Given that this was the first time that this particular set of measures was being collected, there was interest in learning about the feasibility, relevance and usefulness of these measures.  Qualitative data collection in the form of interviews and focus groups were done with a variety of stakeholders, ranging from the people who defined and calculated the measures to the providers who were seeing measure results specific to their site.

Lessons Learned:

  • Do your homework ahead of time – When talking to people involved in the nitty-gritty of data measurement and calculation, you need to have a solid understanding of the technical aspects involved so that you don’t spend the entire interview asking background questions.  Be comfortable with the subject matter.
  • Be flexible – The measure development process takes time. There can be unanticipated challenges or circumstances that can delay any component of the project. If interviews are planned for the end of a particular component, be flexible with the timing.
  • Orient interviewees, if necessary – Not all stakeholders, particularly consumers, will have a strong understanding of what a measure is.  In order to get the desired feedback, you may need to spend time providing some background and education before you start asking questions.

Hot Tips:

  • In a project that takes place over the course of a number of years with several different components, try to complete interviews when the information will be the most current for the interviewee and you.
  • Have a tracking mechanism set up to help you stay organized with your data collection. For us, this takes the form of an Excel spreadsheet containing fields such as interviewee contact information, dates of contact and data collection, and staff responsible for transcription and quality assurance.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

No comments yet.

Leave a Reply




To top