Mel Mark here, recently retired from Penn State. I’ve long been involved in research on evaluation (RoE). In this post, I want to encourage collaboration between evaluation practitioners and RoE-ers (“rowers,” my term for those who do RoE).
First, though, consider the potential value of RoE as shown by a pair of historical examples. In early research on evaluation use, Patton and his colleagues identified the “personal factor:” If a specific individual or individuals, other than the evaluator, was committed to the evaluation and interested in its findings, use was more likely. Ever since, the personal factor has been foundational to Patton’s influential utilization-focused evaluation.. Other early RoE led to Weiss’s notion of enlightenment, a precursor of conceptual use. When enlightenment occurs, evaluation findings lead, not to some immediate decision or action, but to a changed understanding. Of course, not every instance of RoE will have the substantial, long-lasting impact of these early examples. Still, RoE can be a valuable guide to practice.
Now, consider possible collaborations between evaluation practitioners and RoE-ers. Imagine there’s some new procedure that could be used in evaluation. Let’s say it involves new practices for getting input from program clients and other interested parties. Today, the developer of a new procedure might disseminate it via presentations, posts, and articles, perhaps including a description of how well it worked in one or two of their own evaluations. Now imagine that same protocol being implemented in several different evaluations, of a variety of programs, with each evaluator carrying out the data collection about the new procedure, and with each evaluator approving the interpretations made in a common report.
This more collaborative approach would have several benefits. We wouldn’t rely solely on reports from the procedure’s developer. More evidence about generalizability would exist, showing how well the procedure works across different settings, with varied kinds of programs, etc. Another benefit, especially if this kind of collaboration became routinized, would be an increase in RoE grounded in actual evaluation practice. RoE based on surveys of evaluators or lab/simulation studies can be quite valuable, but we need more RoE conducted within real evaluations. Yet another benefit could emerge, especially if the collaboration between a set of RoE-ers and practitioners continues over time. That is, the questions examined in RoE could come from a group of full-time evaluation practitioners, rather than only from RoEers.
I’ll admit that developing these collaborations may not be easy. But some approaches might be easier than others. Perhaps larger graduate programs in evaluation could invite their students and alums to participate. Or larger evaluation firms could see if their investigators were interested. Maybe AEA TIGs could offer a collaborative project. One-off collaborations could be initiated among people with shared interests. And while more challenging to maintain, it would be great to have sustained research-practice partnerships (RPPs), with RoE-ers and evaluation practitioners working over the long-term, and with practitioners having major voice in formulating research questions.
There’s more that could be said about such collaborations. But these posts are short. I hope conversation – and action – ensues!
Rad Resource
Research-practice partnerships in education: The state of the field. This report overviews the status of RPPs in education, with an influential definition of RPPs, examples illustrating the variability across RPPs, and important issues for consideration.
The American Evaluation Association is hosting Research on Evaluation (ROE) Topical Interest Group Week. The contributions all this week to AEA365 come from our ROE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Hi.
This is an exciting agenda of building a network to Collaborate for Research on Evaluation. I imagine this would help to develop evidence base models and methodological approaches to conduct evaluations differently.