{"id":30130,"date":"2024-10-03T02:00:00","date_gmt":"2024-10-03T06:00:00","guid":{"rendered":"https:\/\/aea365.org\/blog\/?p=30130"},"modified":"2024-10-03T00:49:42","modified_gmt":"2024-10-03T04:49:42","slug":"quantitative-methods-theory-and-research-tig-week-statistical-mediation-and-moderation-in-program-evaluation-two-key-principles-evaluators-should-use-by-antonio-olmos","status":"publish","type":"post","link":"https:\/\/aea365.org\/blog\/quantitative-methods-theory-and-research-tig-week-statistical-mediation-and-moderation-in-program-evaluation-two-key-principles-evaluators-should-use-by-antonio-olmos\/","title":{"rendered":"Quantitative Methods: Theory and Design TIG Week: Statistical Mediation and Moderation in Program Evaluation: Two Key Principles Evaluators Should Use by Antonio Olmos"},"content":{"rendered":"\n

Hi! My name is Antonio Olmos<\/strong>, Executive Director of the Aurora Research Institute (ARI) in Aurora, Colorado. ARI, a subsidiary of Aurora Mental Health and Recovery, provides evaluation services in Mental Health, Substance Abuse, Law Enforcement (Crisis Response Teams), and Public Health. We have extensive experience with federal and state grants.<\/p>\n\n\n\n

When we conduct evaluations, we always develop or revise the theory of change and logic models to understand how programs achieve their goals. In doing so, we\u2019ve found that mediation<\/strong> and moderation<\/strong> are essential in uncovering the mechanisms driving program success. However, we rarely see these concepts in other evaluations and wonder why.<\/p>\n\n\n\n

What Are Mediation and Moderation?<\/strong><\/h5>\n\n\n\n

Mediation<\/strong> occurs when a variable (called a mediator) explains the relationship between treatment and outcome. For example, a program designed to improve mental health might lead to increased social support, which then improves outcomes. Social support is the mediator. A mediation model looks like this:<\/p>\n\n\n\n

\"Mediation<\/figure>\n\n\n\n

By identifying mediators, evaluators can better understand how\/why<\/em> a program works. If a program falls short of expectations, mediation analysis may reveal where the breakdown occurs.<\/p>\n\n\n\n

Moderation (<\/strong>often referred to as “interaction effects<\/em>“), on the other hand, affects the strength or direction of the treatment-outcome relationship. For instance, a substance abuse program may work better for younger participants than older ones. Here, age is a moderator. A moderation model looks like this:<\/p>\n\n\n\n

\"Moderation<\/figure>\n\n\n\n

Understanding moderation helps evaluators see who benefits most from a program and under what circumstances.<\/p>\n\n\n\n

Hot Tips<\/strong><\/h4>\n\n\n\n
Using Mediation and Moderation in Evaluation<\/strong><\/h5>\n\n\n\n
    \n
  1. Start with Your Logic Model<\/strong>: Use it to hypothesize potential mediators and moderators. Ask your program stakeholders: What mechanisms lead to change? Who benefits most?<\/em><\/li>\n\n\n\n
  2. Collect the Right Data<\/strong>: Your data collection plan should include measures of mediators (e.g., behaviors, attitudes) and moderators (e.g., demographic characteristics). Ensure your design allows for testing these relationships.<\/li>\n\n\n\n
  3. Use Simple Diagrams<\/strong>: Communicate complex ideas to non-technical stakeholders with clear, simple mediation and moderation diagrams. Simple diagrams can help bridge this gap and make the evaluation findings more accessible.<\/li>\n\n\n\n
  4. Interpret with Caution<\/strong>: While mediation and moderation offer valuable insights, they don\u2019t necessarily establish causality. Just because a variable mediates or moderates an effect doesn\u2019t mean a causal relationship. Make sure claims are supported by rigorous testing.<\/li>\n<\/ol>\n\n\n\n

    Lessons Learned<\/strong><\/h4>\n\n\n\n

    At ARI, incorporating mediation and moderation in evaluations has deepened our understanding of program impacts. We can move beyond simple cause-effect models to explore how and why programs succeed or fail. The challenge often lies in communicating these advanced concepts to stakeholders, but with clear explanations grounded in real-world examples, we\u2019ve found success.<\/p>\n\n\n\n

    Rad Resources<\/strong><\/h4>\n\n\n\n