Hello! We are Kelly Murphy, Doctoral Candidate in Applied Developmental Psychology at Claremont Graduate University (CGU) and Program Chair-Elect of the PreK-12 Educational Evaluation TIG,and Michelle Sloper, Doctoral Student in Positive Developmental Psychology and Evaluation Associate at Claremont Evaluation Center (CEC). With backgrounds in developmental psychology and program evaluation, we have always been interested in how to accurately measure participant changes in intended outcomes over time in our evaluations. Measuring change is particularly tricky when working with youth programs because childhood and adolescence are periods of rapid development. Oftentimes we rely on pre-post test designs that assume that (1) all participants are changing in the same direction and at the same rate and (2) that change is linear (only increasing or decreasing over time). However, these designs and the accompanying analytic techniques do not allow evaluators to accurately answer the following questions:
- Are there individual differences in participants’ change in outcomes?
- Is participant change non-linear?
- What participant characteristics are associated with the greatest improvements or declines, in outcomes?
- Does participation in one program component produce a greater rate of change in participant outcomes than another component?
- Is change in one domain (e.g., non-cognitive skills) associated with change in another domain (e.g., academic achievement)?
Today, we’d like to share some hot tips on assessing longitudinal change in outcomes and provide some rad resources about how to sensitively assess change over time in your own evaluations.
Hot Tip: Design your evaluation to include three points of data collection and conduct latent growth curve models.
Latent Growth Curve (LGC) Modeling:
What is it?
LGC Modeling is considered one of the most powerful and informative methods for assessing change over time. LGC modeling frameworks allow you to capture both non-linear change, and inter-individual differences in change over time that traditional repeated measures statistical strategies fail to capture.
How do we conduct it?
Because LGC modeling is too dense to cover in a single blog post, we have put together some rad resources to help you learn more about LGC modeling.
Rad resources: If you’re interested in learning more about LGC modeling you can find a gentle introduction here and a more comprehensive introduction here. Barbara M. Byrne also has an incredibly accessible SEM textbook with a chapter on LGC modeling.
Hot Tip: Dig deeper into the program’s theory of change to collect data on relevant participant characteristics, experiences, and program dosage. Use these meaningful mediators and moderators to predict differences in change over time in LGC models to clarify the links between program participation and outcomes.
The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.