What is evaluation? And how does it differ from research? by Dana Wanzer

Hi! I’m Dana Wanzer, doctoral candidate at Claremont Graduate University and an avid #EvalTwitter user!

Many people new to evaluation—students and clients alike—struggle with understanding what evaluation is, and many evaluators struggle with how to communicate evaluation to others. This issue is particularly difficult when evaluation is so similar to related fields like auditing and research.

There are many great resources on what evaluation is and how it differs from research, including John LaVelle’s AEA365 blog post, Sandra Mathison’s book chapter, and Patricia Rogers’ Better Evaluation blog post.  I wanted to examine these findings more in depth, so I conducted a study with AEA and AERA members to see how evaluators and researchers defined program evaluation and differentiated evaluation from research.

In this study, I recruited members of AEA (who were primarily members of the PreK-12 Educational Evaluation and Youth-Focused Evaluation TIGs) and members of Division H (Research, Evaluation, and Assessment in Schools) of the American Educational Research Association (AERA). A total of 522 participants completed the survey which, among some other questions, had them define evaluation, choose which of the below diagrams matches their definition of evaluation, and rate how much evaluation and research differs across a variety of study areas (e.g., purpose, audience, design, methods, drawing conclusions, reporting results).

Lesson Learned: Evaluators and researchers alike mostly define evaluation like Scriven’s definition of determining the merit, significance, or worth of something—essentially, coming to a value judgment. However, many evaluators also think evaluation is about learning, informing decision making, and improving programming, indicating the purpose of evaluation beyond simply the process.

Lesson Learned: Mathison described five ways in which evaluation and research could be related:

How evaluation and research are related diagram
(Click for larger image)

Half of participants thought research and evaluation overlap like a Venn diagram, which is similar to the hourglass model from LaVelle’s blog post, and a third thought evaluation is a sub-component of research. However, evaluators were more likely to think research and evaluation intersect whereas researchers were more likely to think evaluation is a sub-component of research. Evaluators are seeing greater distinction between evaluation and research than researchers are!

Lesson Learned: Participants agreed that evaluation most differs from research by the purpose, audience, providing recommendations, disseminating results, and generalizing results and are most similar in study designs, methods, and analyses. However, more evaluators thought evaluation and research differed greatly across a multitude of study-related factors like these compared to researchers.

Hot Tip: If your students or clients already know about research, describing how evaluation is similar to and different from research might be a great approach for teaching what evaluation is!

I believe this study will be useful in helping propel the field of evaluation forward, at least by helping our field better describe evaluation but potentially in situating our field as a distinct discipline and profession.

Rad Resource: Hungry for more information? All the study materials, data, and manuscript are posted on the Open Science Framework, a free and open-source website that allows scientists to collaborate, document, and share research projects, materials, and data.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

17 thoughts on “What is evaluation? And how does it differ from research? by Dana Wanzer”

  1. Heather Keeley

    Hello Dr. Wanzer,

    My name is Heather Keeley and currently I am a graduate student. I am learning the difference between research and evaluation for the second class. We are now going more into the process, but what was strange for me was that before graduate school, I thought I knew the difference. What has happened is that I have gotten more confused. Confusion just means that my perspective is changing and I am being challenged!

    I wanted to first tell you how much this blog post has helped identify my own beliefs with research and evaluation. I looked at Mr. Lavelle’s blog post, and like him I am a very visual learning. The hourglass resonated with what I believe for research and evaluation based off personal experience and the textbook for our class “Research Methods in Education and Psychology: Integrating Diversity with Quantitative and Qualitative Approaches” by Donna Mertens. I believe that they overlap in similarity, but ultimately it is about the stakeholders involved. I am starting to see that there is more stake in evaluation than in research. I teach middle school, so how I am connecting this is to “bullying.” I was bullied, so I see that my own “merit,” “value,” and “worth” were based on those who were evaluating me. I listened to their “results” and changed to fit their definition, so that way I could be more accepted and effective at life. If they spent more time getting to know me and doing their research on me, the results would have changed, but if they just researched me with no evaluation, it would not have affected me. I would not have known their opinions on how to be better. It is not exactly the best metaphor, but it is how I am connecting to it personally.

    I have been researching myself during this quarantine and evaluating my selfworth based on old information and facts. It is time I evaluatd from a different perspective. When my professor asked which paradigm I think is important, I picked constructivist. I have to connect personally to what I am learning and create a narrative to learn. Connecting it to the job is just a little harder currently because my job is not about learning, but completion right now. The skills my students are learning right now are the ones not taught in the classroom.

    Thank you for taking the time to read my input on your blog post! Thank you for giving me an insight into this definition and helping with the clarification process!

  2. Dr. Wanzer,

    I am a graduate student in Learning Design and Technology at Arizona State University. Your article was very helpful in defining research vs evaluation. I initially thought that both terms were related. Upon reading your article and exploring the links provided, I think I have a better definition of what the two terms mean. I originally thought that they were the same because they both deal with identifying data and information. As an experienced evaluator, do you think that research and evaluation is necessary?

    Thanks again for your insight.

    Sincerely,
    Carl

    1. Hi Carl,

      I don’t think I’d be in this profession if I didn’t think that research and evaluation is necessary! How else do we know what works and for whom? How else do we expand the knowledge base? Research and evaluation help us in both endeavors.

      -Dana

  3. Dr. Wanzer,

    I am studying Curriculum and Instruction at Arizona State University and currently enrolled in a research course. I really appreciated your article and visuals you had to help explain the five ways research and evaluation could be related. I also appreciated the links to other pages that support your information and findings on research and evaluation. Thank you for the resources, such as Open Science Framework. Do you believe that evaluators are seeing the difference between the two more than researchers are?

    Thank you again!

    Lauren Foley

    1. Hi Lauren,

      Yes, my research found that evaluators see more differences between research and evaluation than researchers do. In other words, we see the uniqueness of our work more than researchers do.

      -Dana

  4. Paige Russell

    Dr. Wanzer,
    I’m enrolled in a graduate course at Arizona State University that is currently discussing the differences between research and evaluation. Upon searching for resources to inform my discussion post, I came across your article. Thank you for including helpful links, diagrams, and resources. I found it fascinating that evaluators are more likely to view evaluation and research as only intersecting whereas researchers leaned more towards evaluation as a sub-component of research. Before taking this course and learning about these differences, I honestly thought they were one and the same. Now that I know how evaluation goes beyond the analysis of data into determining value (worth and merit) of the findings, I feel as though I agree with the comment above mine as research and evaluation being cyclical in nature with both research and evaluation informing one another. I’m a classroom teacher and I have definitely had my students evaluate their findings but used the verbiage of research! Now that I can distinguish between the two, I want to further my own understanding on the differences and pass that along to my own students.
    Thank you again for your wonderful article- I found it to be incredibly helpful and straightforward.
    Sincerely,
    Paige

  5. Jennifer Baron

    Dr. Wanzer,

    I couldn’t agree more with your claim that many people new to evaluation struggle to understand it and explain the relationship it has to research. I am a middle school teacher pursuing a graduate degree in Learning Design and Technologies. It wasn’t until I started taking graduate courses that I even tried to tackle the concept of evaluations. As a teacher, and a pretty new one at that, I am used to assessing students and having administrators evaluate me, so I’ve never really done the evaluating myself. This makes it very difficult for me to fully understand what evaluations are, what they involve, and how they relate to research. However, after reading what you have to say about it, combined with the text I read for my course, I have a much better grasp of it.

    The graphic you included was very helpful when it came to my perception of evaluation relating to research. I can see how all five are plausible relationships and I think they can change depending on the content that is being researched and evaluated. Because of this, I tend to gravitate toward the idea that they exist on a continuum.

    Something I’ve been thinking about is whether or not research can be effective without an evaluation component and vice versa. I would love to hear your thoughts on this because I’m having a hard time deciding. I am inclined to think that research can be effective without an evaluation, but an evaluation cannot be effective without research. However, I don’t know if this is an accurate claim or if I’m considering all possibilities. I would appreciate your perspective on this idea.

    Thank you for your valuable input on this topic.

    Jennifer Baron

    1. Hi Jennifer,

      Great questions! I think they’re more synergistic that many believe. We see research being used in evaluation a lot, especially in theory-driven evaluation. However, I think we should also be thinking about research on evaluation to inform our evaluation practices.

      The incorporation of evaluation into research may be less obvious, especially with research tending to want to remain ‘value-neutral.’ Yet researchers embed their values into their research when they choose what and who to study, with what methods, with what data analysis tools, etc. (these might be the “merit” of a study). Approving a study for a grant or a paper for publication indicate the value we place on research (these might be the “worth” of a study). And statistical significance and effect sizes indicate the value we place on the findings (these might be the “significance” of a study). Research has evaluation all throughout it, which is why we need to embrace and advocate for the role of evaluation as a transdiscipline (and perhaps, as Scriven argues, the alpha discipline).

      -Dana

  6. Jackie Arthur

    Hi Dr. Wanzer,

    I am currently in a graduate program that focuses on learning design and technology, which involves me taking courses on both research and evaluation. I would definitely consider myself a novice when it comes to evaluation considering I have only taken one formal course on the topic and in my career life, I have evaluated learning programs, but not to the scope of a true evaluation.

    I would have to say that study you conducted and the topic mentioned is extremely common since I really do struggle to this day to explain the difference between research and evaluation. I know from my courses and experience that they’re not one in the same, but it does become difficult to explain the differences to others. I would say I connect with the participants that believe research and evaluation are similar in scope to a venn diagram. I have always felt that they overlap in some regard because there is a bit of research necessary in the evaluation process to either understand the program being evaluated or the company being evaluated. I believe that research can further lift up the evaluators in their endeavor to evaluate.

    I also will say that it is important to distinguish the difference between evaluation and research when working with a client or organization so that they understand the scope of exactly what you are there to do as an evaluator. Your tips are extremely helpful!

    Thanks for your insights and even the graphics help explain the different beliefs around this topic even further.

    Sincerely,
    Jackie Arthur

  7. Heather Allen

    Dr. Wanzer,
    I wanted to let you know how much I enjoyed this article. I’m a graduate student and between my recent class about evaluation and current class about research, this topic is coming up a lot for me. Until I took the course on evaluation last semester I thought evaluation and research were basically the same things. I know better now. This article added more to my knowledge in this area by pointing a lens at the evaluator’s perspective. I think it is a great perspective to have. Thank you for including the links to the data, diagrams, and other information of the study. I downloaded all of them and, as a beginner to the field of evaluation loved seeing the things I was taught in my class “come to life” in a real-life scenario. I thought the data in your figures and tables were easy to understand and the results are enlightening.

    As I am a novice in the field of evaluation, do you have any textbooks or other books you suggest to teach someone about evaluation? Any favorite resources you use (blogs, books, other publications) that you consider “must-read” information to stay up-to-date in the world of evaluation? I’ve already read the article from Patricia Rogers suggested in your post and will read the others you list above as well but I’m very interested in both Research and Evaluation and would appreciate any suggestions you might have from your experiences.

    Thank you!

    1. I find AEA365, #EvalTwitter, and following my favorite evaluation journals (e.g., American Journal of Evaluation, Canadian Journal of Program Evaluation, Evaluation and Program Planning, Journal of MultiDisciplinary Evaluation) the best way to stay up-to-date in the world of evaluation! There are a ton of resources out there, and the community is always sharing in a wide variety of mediums including blogs, podcasts, journal articles, webinars, and more.

  8. Pingback: #EvalTuesdayTip: Identifying relationships between evaluation & research - Khulisa

  9. Erika Stanischewski

    Hi Dana,

    Your post is of particular interest since it has prompted me to reflect on the relationship between evaluation and research within the context of teaching. I am a high school science teacher, and I have always regarded myself primarily as a deliverer of information that I hoped would become knowledge to the receiver. I did not consider myself an evaluator at the same time, and it was only until recently that my thought process began to change.

    When reading your post, I was immediately inclined to support the Venn diagram, and thus the hourglass model that LaVelle suggested in his post, to represent my idea of how evaluation and research are related. I can appreciate that research and evaluation both have their respective purposes, research being to discover “generalizable knowledge” and evaluation to discover “context-specific knowledge” (Alkin, M. C. & Taut, S. M., 2003), but that they share common methodologies and analysis techniques to render their conclusions.

    However, when I think of my role as a teacher, I see myself as using the results of an evaluation to illicit more research, to then test out and re-evaluate. I see myself learning through this process. When you say, “many evaluators also think evaluation is about learning, informing decision making, and improving programming, indicating the purpose of evaluation beyond simply the process”, it validates my belief that evaluation can elicit more research and can result in learning on behalf of the evaluator for the benefit of the stakeholders: my students.

    For example, I am evaluating whether a specific set of laboratory skills have been internalized by my students, and whether they are then transferrable to another scenario. Through the evaluation, I discover that the transfer of knowledge did not occur. Therefore, I research different methods for ensuring transferability of knowledge and then try the evaluation again to see whether my new approach has worked. Within my context of teaching, I am seeing evaluation and research as cyclical in that there really is no beginning or end; whether we begin with research that we then evaluate, or an evaluation that leads to more research. Either research or evaluation prompts the other in a cyclical manner. I wonder what your thoughts are in regards to my perspective.

    Thank you for a great thought provoking post, and congratulations on your work in the field of evaluation.

    1. I definitely agree that research and evaluation can and should be a cyclical process! I think we think of research informing evaluation more so than the other way around, but they can both improve how we think about the other. Research can inform us on what might be useful programs or interventions, but evaluations can elucidate things that might need further research to determine whether it can generalize to other people, situations, contexts, etc.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.