Hello! We are Stacy Johnson and Cami Connell from the Improve Group. At Evaluation 2013, we had the opportunity to present on our experiences using a unique mixed methods approach to collecting data.
Your data collection strategy has the potential to seriously impact your evaluation. You might ask yourself questions like: How do we make sure we are getting the whole story? What if one method isn’t appropriate for gathering all the information you need from a single source? How do you engage people in data collection in a way that makes them understand and want to use the findings? One way to address these questions is to think about each stage of data collection as a layered process by directly connecting quantitative and qualitative methods to complement each other and build a more in-depth and accurate story.
How is this different from how we traditionally think about data collection? We still access the same key sources to answer our evaluation questions, but the design includes a feedback loop to allow the evaluator to immediately integrate any initial findings into the data collection process as they emerge. This often means intentionally including additional interviews or focus groups after an initial stage of data collection to present data back to stakeholders and ask for feedback and relevant background about emerging themes.
Lesson Learned: Provide an orientation to data. Not everyone looks at data every day! Walking stakeholders through data increases the chances that they will want to use it to inform decisions.
Hot Tip: Create easy to interpret graphics to make data more accessible.
Lesson Learned: Make it a mutually beneficial process. In addition to gathering important information for the evaluation, it is equally important to make sure people feel like they are heard and that sharing their experiences can positively impact their work.
Hot Tip: Facilitate discussion about how data applies in day-to-day work.
Hot Tip: Encourage problem solving and planning for how data can inform changes or improvements.
Lesson Learned: Understand the stakes and relationships. Depending on the nature of relationships and potential consequences of the evaluation, there is a risk of people painting an overly positive or overly negative picture. In addition, when presenting data from one source to another, careful attention should be paid in masking the identity of the original source, especially when there are easily identifiable groups or an existing adversarial relationships.
Hot Tip: Include people with different perspectives and roles in the data collection process to uncover any underlying dynamics.
Hot Tip: Try to be aware of any adversarial or contentious relationships that may exist. This approach is not always appropriate depending on existing relationships.
Hot Tip: Mask the original source of data as appropriate.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.