Hello! My name is Dana Linnell (formerly Dana Wanzer). Along with my colleagues Esther Nolton, Michael Harnar, Seema Mahato, and Travis R. Moore, we are going to share information about research on evaluation (RoE) presented at the American Evaluation Association conference.
For context, here is the definition of RoE that guided our work: “Any purposeful, systematic, empirical inquiry intended to test existing knowledge, contribute to existing knowledge, or generate new knowledge related to some aspect of evaluation processes or products, or evaluation theories, methods, or practices.” (Coryn et al., 2017, p. 332).
We often assume that RoE is typically found in peer-reviewed journals, such as the American Journal of Evaluation. In fact, a research study conducted in 2017 by individuals affiliated with Western Michigan University examined a decade’s worth of articles published in 14 evaluation-related journals. They discovered that 7.6% of those articles focused on RoE. The definition above was used to screen articles for whether or not they were RoE.
We often hear that publishing in peer-reviewed journals is challenging for individuals outside of academia due to the time commitment and lack of incentives to publish. Since the field of evaluation is primarily practice-oriented, with only 26% of AEA members working in academia and a mere 10% identifying themselves primarily as researchers in 2018, it is essential for us to explore beyond publications and academic spaces to discover relevant RoE studies and findings.
So who is doing research on evaluation, and, if it is not being published in peer-reviewed journals, where might it be? We argue that evaluation practitioners are doing research on evaluation to some extent, and that it is more likely to be shared at the annual American Evaluation Association conferences rather than in academic journals. To examine this, we gathered all accepted proposals from the 2019 AEA conference and coded them for whether they were research on evaluation (i.e., whether proposals were systematic, empirical, and focused on evaluation).
Lessons Learned
We found that out of the 1539 accepted proposals that we examined, 108 (14.7%) proposals at the 2019 AEA conference were research on evaluation. Most of these were in multipaper (26.1%) or paper (17.1%) sessions compared to ignite (13.0%) or poster (7.7%) sessions (note: we did not analyze the other session types because we did not think they were likely to include RoE studies).
Unsurprisingly, the RoE TIG had the highest percentage of proposals that were RoE (71%). There were four proposals that were not RoE: the first described evaluations and was not a systematic research study on evaluation, the second discussed how to integrate RoE into evaluation practice, and the third and fourth proposals described topics that could have been research but there was not sufficient information on the data or methods to determine whether it was systematic or empirical.
However, other TIGs also had a high percentage of RoE submissions, including social impact measurement (40.0%), teaching of evaluation (40.0%), design and analysis of experiments (37.5%), extension education evaluation (35.7%), use and influence of evaluation (35.3%), indigenous people in evaluation (33.3%), quantitative methods (33.3%), and theories of evaluation (30.0%).
Most of the proposals that we coded as not RoE were because they were presenting an evaluation (not research ON evaluation) or presenting research on some other topic. Many other proposals were not systematic or not empirical, meaning they were reflections without systematic methods or did not use any data or observations as part of their proposals.
Rad Resource
If you’d like to learn more about RoE, check out the RoE TIG’s website and its past webinars and resources linked on the homepage, including our webinar on Getting Started with Research on Evaluation.
The American Evaluation Association is hosting Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.