Stacy Johnson on Analyzing Less Than Perfect Longitudinal Data

Hi, I’m Stacy Johnson, Senior Research Analyst at the Improve Group a national and international evaluation consulting organization based in Saint Paul, MN. The Improve Group works with organizations to make the most of information, navigate complexity and ensure their investments of time and money lead to meaningful, sustained impact. This past October at Evaluation 2012, I had the opportunity to present on my experiences analyzing longitudinal data with less than perfect data.

Being thoughtful in the planning and data collection process plays a crucial role in successfully collecting the data you need to address your evaluation questions. But, what happens if you are not involved in this process? The first step is to assess what you have to work with and figure out how to move forward. Sometimes you may be pleasantly surprised at what a great job was done in gathering data over time. Other times you need to figure out how to make the best out of what you have and focus on the control you have over what happens from this point forward. This aea365 includes some of the lessons I have learned and tips that helped me along the way.

Lesson Learned: Changes are often made to data collection tools over time including changes to wording of items, changes to response options, and eliminating and adding items.

Hot Tips:

• Recommend selecting key variables to track and keep consistent
• Explain implications of making changes
• Facilitate thinking ahead
• Create a database to use as a guide for what should be tracked

Lesson Learned: The process of collecting data can be unclear with data collected by a variety of people.

Hot Tips:

• Create a formal process with detailed instructions and protocols
• Facilitate clear communication and training of data collectors

Lesson Learned: Messy datasets! Data is collected in different formats, is poorly cleaned, incorrectly merged, there are no clear data cleaning decisions, and there are unknown variable names, labels, and coding.

Hot Tips:

• Request original data
• Talk to those involved about the decisions that were made
• Get copies of instruments

Lesson Learned: There is a need to manage unrealistic expectations when you are asked questions the data cannot answer.

Hot Tips:

• Discuss what can be shown (and what cannot) with the data collected
• Balance expectations with reality – you may need to guide others on how feasible their requests are and the limitations of the data
• Facilitate thinking ahead (again) – help others think about their future evaluation needs and how their work may evolve over time

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.