Stacy Johnson on Analyzing Less Than Perfect Longitudinal Data

Hi, I’m Stacy Johnson, Senior Research Analyst at the Improve Group a national and international evaluation consulting organization based in Saint Paul, MN. The Improve Group works with organizations to make the most of information, navigate complexity and ensure their investments of time and money lead to meaningful, sustained impact. This past October at Evaluation 2012, I had the opportunity to present on my experiences analyzing longitudinal data with less than perfect data.

Being thoughtful in the planning and data collection process plays a crucial role in successfully collecting the data you need to address your evaluation questions. But, what happens if you are not involved in this process? The first step is to assess what you have to work with and figure out how to move forward. Sometimes you may be pleasantly surprised at what a great job was done in gathering data over time. Other times you need to figure out how to make the best out of what you have and focus on the control you have over what happens from this point forward. This aea365 includes some of the lessons I have learned and tips that helped me along the way.

Lesson Learned: Changes are often made to data collection tools over time including changes to wording of items, changes to response options, and eliminating and adding items.

Hot Tips:

  • Recommend selecting key variables to track and keep consistent
  • Explain implications of making changes
  • Facilitate thinking ahead
  • Create a database to use as a guide for what should be tracked

Lesson Learned: The process of collecting data can be unclear with data collected by a variety of people.

Hot Tips:

  • Create a formal process with detailed instructions and protocols
  • Facilitate clear communication and training of data collectors

Lesson Learned: Messy datasets! Data is collected in different formats, is poorly cleaned, incorrectly merged, there are no clear data cleaning decisions, and there are unknown variable names, labels, and coding.

Hot Tips:

  • Request original data
  • Talk to those involved about the decisions that were made
  • Get copies of instruments

Lesson Learned: There is a need to manage unrealistic expectations when you are asked questions the data cannot answer.

Hot Tips:

  • Discuss what can be shown (and what cannot) with the data collected
  • Balance expectations with reality – you may need to guide others on how feasible their requests are and the limitations of the data
  • Facilitate thinking ahead (again) – help others think about their future evaluation needs and how their work may evolve over time

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

3 thoughts on “Stacy Johnson on Analyzing Less Than Perfect Longitudinal Data”

  1. Hi Stacy!

    We just finished Year 1 of a longitudinal study, and we are anticipating some of these data issues you brought up in your post–so this was helpful to read! Are there any resources you recommend as we prep for Year 2? Thanks!

  2. The world of data management is taking of at an amazing pace. If you are advising young professionals, it’s a smart career to take on. Data will not being going away nor will it become less complicated. We need more professionals sensitized to the organizational needs of data in the social sciences.

    IASSIST is an international organization of professionals working in and with information technology and data services to support research and teaching in the social sciences. It’s a great place to get started or accelerate a career in data management. Here’s the link to this year’s conference site: “Data Innovation: Increasing Accessibility, Visibility and Sustainability”

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.