Hello, we are Linda Cabral and Jillian Richard-Daniels from the Center for Health Policy and Research at University of Massachusetts Medical School. Collecting and reporting data on a set of predefined measures is something that many evaluators are asked to do. This typically quantitative process involves gathering data, often from multiple sources and stakeholders, to assess the amount, cost, or result of a particular activity. But have you ever thought about what goes into measure development, collection and reporting? A recent evaluation that we completed included interviews with the people involved in this process.
A federal grant interested in testing a set of child health quality measures was awarded to a group of stakeholders in Massachusetts. An example of such a quality measure is the percentage of infants who reached age 15 months during the measurement year, and who has six or more well infant visits during the first 15 months of life. Given that this was the first time that this particular set of measures was being collected, there was interest in learning about the feasibility, relevance and usefulness of these measures. Qualitative data collection in the form of interviews and focus groups were done with a variety of stakeholders, ranging from the people who defined and calculated the measures to the providers who were seeing measure results specific to their site.
Lessons Learned:
- Do your homework ahead of time – When talking to people involved in the nitty-gritty of data measurement and calculation, you need to have a solid understanding of the technical aspects involved so that you don’t spend the entire interview asking background questions. Be comfortable with the subject matter.
- Be flexible – The measure development process takes time. There can be unanticipated challenges or circumstances that can delay any component of the project. If interviews are planned for the end of a particular component, be flexible with the timing.
- Orient interviewees, if necessary – Not all stakeholders, particularly consumers, will have a strong understanding of what a measure is. In order to get the desired feedback, you may need to spend time providing some background and education before you start asking questions.
Hot Tips:
- In a project that takes place over the course of a number of years with several different components, try to complete interviews when the information will be the most current for the interviewee and you.
- Have a tracking mechanism set up to help you stay organized with your data collection. For us, this takes the form of an Excel spreadsheet containing fields such as interviewee contact information, dates of contact and data collection, and staff responsible for transcription and quality assurance.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Linda and Jillian,
Thank-you for your post. I am currently learning about program evaluation and data collection. Through my learning, I am creating a program evaluation design that is utilizing qualitative data for analysis. I really appreciate your “lessons learned”. Qualitative data is useful as it allows participants to be exploratory in their responses. Particular responses may be unexpected and certain conclusions may be identified that otherwise may have gone unnoticed if a pure quantitative analysis was utilized. Being well prepared heading into participant interviews seems critical! I wonder if this preparation could be improved by utilizing collaboration between the evaluator and participants? Observation visits would also likely be beneficial. One approach that seems to hone in on these characteristics is the process use model of evaluation. Flexibility also seems necessary. This is something I need to remind myself. I am very task oriented and like to stay on track. I despise falling behind! I have read that qualitative studies tend to take more time. Interviews are lengthy and organizing meetings can likely be challenging. Having a clear process of how to organize data and compile demographic statistics is also a very important piece of any analysis. This article was succinct and has helped me to build an initial framework of how I may go about collecting data. I work in health care so having a perspective from this field was also helpful!
– Matt Kushneriuk