My name is Josh Twomey, Assistant Professor of Family Medicine & Community Health, at UMass Medical School’s Center for Health Policy and Research. Perhaps you have noticed the term ‘Bayesian’ popping up now and then in the evaluators’ break room. I certainly have, and in recent months, set out on a statistical journey to find out why. In this two-part entry, I would like to share some discoveries of this journey. Part 1 includes a basic overview of Bayesian analysis. Part 2 (coming tomorrow) gives an example of how this may benefit our work as evaluators. One note of caution: This may seem less relevant to those doing strictly qualitative data collection and reporting!
Most of us are trained in Null Hypothesis Significance Testing (NHST) whereby we reject a null hypothesis (p < .05) or fail to reject the null (p > .05). This decision is justified by the data we collect, but does not take into account past research findings or expert opinion. Bayesian analysis differs from NHST in that past knowledge is included in our analysis, thereby having a direct influence on our conclusions.
The first step of Bayesian analysis is to quantify this past knowledge via a prior distribution or prior for short. Using priors we are able to specify distribution(s) of parameters (e.g., means, standard deviations) we are interested in. In cases where we have a lot of prior information, we may set up narrow distributions (informative priors). When we do not have a lot of prior information, we may set up very wide distributions (noninformative priors). Priors are used to weight the likelihood of our collected data to produce the posterior distribution. Thus, the posterior is a result of past knowledge updated by our collected data. It is from this posterior where samples can be drawn and conclusions about our evaluations are made.
Hot Tips: In addition to allowing for the use of past knowledge, advantages of Bayesian statistics include:
- Decision-making tools such as Bayes Factors and Highest Density Intervals (HDIs), which can be easier for stakeholders to understand compared to p values and confidence intervals.
- No need to limit the number of hypotheses you wish to test with your data for fear of inflated Type I error (which in my experience can frustrate stakeholders).
- Better capacity to work with Ns that are small, limiting our ability to detect differences/trends, or large where differences may be detected due to large samples, not meaningful differences.
For quick reference/definitions of NHST, p-values, and Bayesian inference click HERE.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.