My name is Josh Twomey, Assistant Professor of Family Medicine & Community Health, at UMass Medical School’s Center for Health Policy and Research. Perhaps you have noticed the term ‘Bayesian’ popping up now and then in the evaluators’ break room. I certainly have, and in recent months, set out on a statistical journey to find out why. In this two-part entry, I would like to share some discoveries of this journey. Part 1 includes a basic overview of Bayesian analysis. Part 2 (coming tomorrow) gives an example of how this may benefit our work as evaluators. One note of caution: This may seem less relevant to those doing strictly qualitative data collection and reporting!
Most of us are trained in Null Hypothesis Significance Testing (NHST) whereby we reject a null hypothesis (p < .05) or fail to reject the null (p > .05). This decision is justified by the data we collect, but does not take into account past research findings or expert opinion. Bayesian analysis differs from NHST in that past knowledge is included in our analysis, thereby having a direct influence on our conclusions.
The first step of Bayesian analysis is to quantify this past knowledge via a prior distribution or prior for short. Using priors we are able to specify distribution(s) of parameters (e.g., means, standard deviations) we are interested in. In cases where we have a lot of prior information, we may set up narrow distributions (informative priors). When we do not have a lot of prior information, we may set up very wide distributions (noninformative priors). Priors are used to weight the likelihood of our collected data to produce the posterior distribution. Thus, the posterior is a result of past knowledge updated by our collected data. It is from this posterior where samples can be drawn and conclusions about our evaluations are made.
Hot Tips: In addition to allowing for the use of past knowledge, advantages of Bayesian statistics include:
- Decision-making tools such as Bayes Factors and Highest Density Intervals (HDIs), which can be easier for stakeholders to understand compared to p values and confidence intervals.
- No need to limit the number of hypotheses you wish to test with your data for fear of inflated Type I error (which in my experience can frustrate stakeholders).
- Better capacity to work with Ns that are small, limiting our ability to detect differences/trends, or large where differences may be detected due to large samples, not meaningful differences.
Rad Resources:
Bayesian Statistics for the Social Sciences by David Kaplan
Doing Bayesian Data Analysis by John Kruschke
For quick reference/definitions of NHST, p-values, and Bayesian inference click HERE.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Pingback: Towards a “post p < 0.05 era” by Tamara Young · AEA365
Thanks Nancy and Elizabeth.
To be honest, I dont think you can go wrong with either book. They were well worth the investment and I refer to them often in my work.
Josh
Nancy- I agree with you. The Kruschke book looks like a great resource (especially since it highlighted on advanced undergrads or first year grad students).
And the post itself was timed perfectly based on a discussion we had at our weekly research/evaluation meeting Thursday. We (and I mean UG and Grad) are taught to reject or fail to reject our null hypothesis, but this is the first time I’ve heard of this approach and I am intrigued by it!! I plan on bringing this up in the next weekly meeting to see if this is something we can be taught (or incorporate) in a future class! Thanks for these two posts Professor Twomey!
Hi Josh – thanks so much for your post! I have been interested in Bayes but overwhelmed at the prospect of learning it. I checked out the Kruschke book on Amazon – it looked great. I don’t use R, JAGS, or Stan, but it sounds as if the book would be worth reading anyway. Do you think that’s true?