I’m Steven J. Pierce, a statistician with a background in community psychology. I’ve been contributing to evaluations for about 20 years, focusing on study design and applying quantitative methods. Over the last few years, I’ve been exploring how evaluators can benefit from adopting some tools and practices for reproducible research.
The foundation of reproducible research is sharing the data and materials other people need to exactly reproduce your findings. Many researchers, scientific disciplines, funders, journals, and organizations have expressed a growing interest in, and emphasis on, reproducibility as a minimum standard for science and as part of the open science movement. Evaluators should be joining this.
Lessons Learned
Enhancing the reproducibility of evaluation reports enacts several of AEA’s guiding principles: Systematic inquiry because it requires meeting high technical standards for conducting analyses, competence because demands learning new skills, and integrity because it increases transparency of evaluation findings.
Achieving reproducibility is a matter of changing our workflows. Replacing manual processes with better, more reproducible approaches to data management and analysis pays big dividends but it requires an investment in learning new tools and ways to work with data.
The quality, accuracy, and efficiency of my analyses improved when I started creating dynamic documents that automate producing fully-formatted reports, manuscripts, websites, or presentation slides complete with text, equations, figures, tables, and reference sections. They also make it easy to update reports quickly.
Following advice from Marwick, Boettiger, and Mullen about using R packages as research compendiums containing dynamic documents, data, and documentation has helped me organize, share, and collaborate on materials. Creating a compendium for each project makes my work easier.
Adopting professional version control tools is worth the effort. It was Bryan’s paper that persuaded me to give that a try.
Rad Resources
I have found that R, RStudio, Quarto, and Git comprise an elegant, powerful, and integrated set of free software tools for doing reproducible analyses and reports.
R is excellent statistical computing software.
RStudio Desktop is a fantastic editor for working with R scripts and Quarto documents that contain R code. It has built-in support for using Git.
Quarto allows you to use R code in dynamic documents that render to many different output formats.
Git is a fantastic software tool for version control, especially on code.
GitHub.com is an online service that facilitates sharing and collaborating on code by hosting Git repositories. You can make repositories public (accessible by everyone) or private (accessible only by collaborators you select).
Hot Tip
You can learn more at AEA 2024: the Quant TIG is hosting a session on “Generating reproducible statistical analyses and evaluation reports: Principles, practices, and free software tools”.
The American Evaluation Association is hosting Quantitative Methods: Theory and Research Evaluation Week. The contributions all this week to AEA365 come from our Quant TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.