This is Josh Twomey again from UMass Medical School’s Center for Health Policy and Research. As promised in yesterday’s Part 1 posting, I wanted to walk through an example demonstrating aspects of Bayesian analysis that evaluators might find advantageous.
Part 1 mentioned the use of the Bayes Factor (BF) as a decision-making tool. BFs tell us the degree one hypothesis is supported by the data in relation to how much another hypothesis is supported by the data. Thus, a BF is an odds ratio showing us which hypothesis is more likely. Now, let’s look at that example.
Suppose you are evaluating the effectiveness of a health psychology program in helping patients manage chronic disease. As part of the evaluation, you measure the self-efficacy of 200 patients managing their diabetes before and after working with a health psychologist. Traditionally, you could do this with a paired t-test of patients’ pre and post self-efficacy scores. As good evaluators, we would begin this analysis with some idea (based on review of similar evaluations or literature) as to the effectiveness of our program. However, in a traditional test, this prior knowledge cannot be factored into our analysis. Our test would produce t and p values such as t = – 3.19, p < .05. With this result we can state post scores are significantly higher than pre scores, but we cannot state the extent to which this conclusion is more likely than our null hypothesis.
With Bayesian analysis, we conduct this paired sample t-test but weight our data by our prior knowledge. For example, if past evaluations tell us that we should expect small effect sizes, we can specify a prior whereby small effect sizes are more likely than larger ones. In this Bayesian framework, a t = -3.19 corresponds to a BF of 10.1. This means that our hypothesis that post scores are different than pre scores is 10.1 times more likely than a hypothesis of no difference.
Hot Tips: The example above highlights 3 advantages of Bayesian analysis:
- Prior knowledge is incorporated into the analysis as our data is weighted by this knowledge;
- We are given direct odds associated with our conclusion; and
- The interpretation of the BF is a clear, intuitive interpretation of our results for stakeholders to understand.
Lessons Learned: Over the course of my journey, which is far from over, I have learned that Bayesian analysis is complex – full of intimidating terms such as conjugate priors and Markov Chains. But considering its advantages, as well as growing demand in our field, I have found the journey to be well worth it.
Rad Resources: Bayesian Factor calculators and literature can be found at: pcl.missouri.edu.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.