Hi, I’m Stacy Johnson, Senior Research Analyst at the Improve Group a national and international evaluation consulting organization based in Saint Paul, MN. The Improve Group works with organizations to make the most of information, navigate complexity and ensure their investments of time and money lead to meaningful, sustained impact. This past October at Evaluation 2012, I had the opportunity to present on my experiences analyzing longitudinal data with less than perfect data.
Being thoughtful in the planning and data collection process plays a crucial role in successfully collecting the data you need to address your evaluation questions. But, what happens if you are not involved in this process? The first step is to assess what you have to work with and figure out how to move forward. Sometimes you may be pleasantly surprised at what a great job was done in gathering data over time. Other times you need to figure out how to make the best out of what you have and focus on the control you have over what happens from this point forward. This aea365 includes some of the lessons I have learned and tips that helped me along the way.
Lesson Learned: Changes are often made to data collection tools over time including changes to wording of items, changes to response options, and eliminating and adding items.
- Recommend selecting key variables to track and keep consistent
- Explain implications of making changes
- Facilitate thinking ahead
- Create a database to use as a guide for what should be tracked
Lesson Learned: The process of collecting data can be unclear with data collected by a variety of people.
- Create a formal process with detailed instructions and protocols
- Facilitate clear communication and training of data collectors
Lesson Learned: Messy datasets! Data is collected in different formats, is poorly cleaned, incorrectly merged, there are no clear data cleaning decisions, and there are unknown variable names, labels, and coding.
- Request original data
- Talk to those involved about the decisions that were made
- Get copies of instruments
Lesson Learned: There is a need to manage unrealistic expectations when you are asked questions the data cannot answer.
- Discuss what can be shown (and what cannot) with the data collected
- Balance expectations with reality – you may need to guide others on how feasible their requests are and the limitations of the data
- Facilitate thinking ahead (again) – help others think about their future evaluation needs and how their work may evolve over time
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.