What is evaluation? And how does it differ from research? by Dana Wanzer

Hi! I’m Dana Wanzer, doctoral candidate at Claremont Graduate University and an avid #EvalTwitter user!

Many people new to evaluation—students and clients alike—struggle with understanding what evaluation is, and many evaluators struggle with how to communicate evaluation to others. This issue is particularly difficult when evaluation is so similar to related fields like auditing and research.

There are many great resources on what evaluation is and how it differs from research, including John LaVelle’s AEA365 blog post, Sandra Mathison’s book chapter, and Patricia Rogers’ Better Evaluation blog post.  I wanted to examine these findings more in depth, so I conducted a study with AEA and AERA members to see how evaluators and researchers defined program evaluation and differentiated evaluation from research.

In this study, I recruited members of AEA (who were primarily members of the PreK-12 Educational Evaluation and Youth-Focused Evaluation TIGs) and members of Division H (Research, Evaluation, and Assessment in Schools) of the American Educational Research Association (AERA). A total of 522 participants completed the survey which, among some other questions, had them define evaluation, choose which of the below diagrams matches their definition of evaluation, and rate how much evaluation and research differs across a variety of study areas (e.g., purpose, audience, design, methods, drawing conclusions, reporting results).

Lesson Learned: Evaluators and researchers alike mostly define evaluation like Scriven’s definition of determining the merit, significance, or worth of something—essentially, coming to a value judgment. However, many evaluators also think evaluation is about learning, informing decision making, and improving programming, indicating the purpose of evaluation beyond simply the process.

Lesson Learned: Mathison described five ways in which evaluation and research could be related:

How evaluation and research are related diagram
(Click for larger image)

Half of participants thought research and evaluation overlap like a Venn diagram, which is similar to the hourglass model from LaVelle’s blog post, and a third thought evaluation is a sub-component of research. However, evaluators were more likely to think research and evaluation intersect whereas researchers were more likely to think evaluation is a sub-component of research. Evaluators are seeing greater distinction between evaluation and research than researchers are!

Lesson Learned: Participants agreed that evaluation most differs from research by the purpose, audience, providing recommendations, disseminating results, and generalizing results and are most similar in study designs, methods, and analyses. However, more evaluators thought evaluation and research differed greatly across a multitude of study-related factors like these compared to researchers.

Hot Tip: If your students or clients already know about research, describing how evaluation is similar to and different from research might be a great approach for teaching what evaluation is!

I believe this study will be useful in helping propel the field of evaluation forward, at least by helping our field better describe evaluation but potentially in situating our field as a distinct discipline and profession.

Rad Resource: Hungry for more information? All the study materials, data, and manuscript are posted on the Open Science Framework, a free and open-source website that allows scientists to collaborate, document, and share research projects, materials, and data.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

35 thoughts on “What is evaluation? And how does it differ from research? by Dana Wanzer”

  1. Dr. Wanzer,

    Thank you for this post about people views of research and evaluation. I am a graduate student at ASU and currently taking my first class related to research and evaluation, so your post really helps! It can be hard to distinguish between the differences of research and evaluation. Personally, up until a few days ago I thought the terms were synonymous. Your post helped to identify the differences between the two, although they are closely related. What I am still wondering if the two processes can be effective when applied separately, or if both need to be used. Again, thank you for an insightful article!

  2. Hi Dana,

    I have recently began my study of research and evaluation as part of my master’s degree pursuit at Arizona State. I appreciated the graphic of the 5 different thoughts of how research and evaluation may be connected. I do wonder if if the two have a symbiotic nature. For example, research may lead to new programs or systems, which are then evaluated for effectiveness, resulting in new research opportunities based on their effectiveness. I think I see overlap in the disciplines, but I am wondering if this line of thought goes with the continuum theory you showed in that graphic.

  3. Dr. Wanzer,

    Thank you for sharing the interesting results of your study. Prior to beginning specific inquiry into the fields of research and evaluation, I was unaware of the debate and lack of consensus on the two topics. Quite frankly, I am a bit shocked at the collective effort that has been focused on finding new and unique ways of describing the relationship between the two.

    You mention in your closing thoughts the possibility of evaluation moving towards existing as a separate profession. What do you believe needs to occur to move in that direction? Would there be any negative ramifications for either the research or evaluation field if evaluation did evolve into a unique and distinct discipline?

    Best,

    Stacey

  4. Hello Dr. Wanzer!
    I am a graduate student at Arizona State University studying curriculum and instruction for gifted education. I find research and evaluation fascinating, but from reading your article I have found that they are not as related as I thought they were. I firmly believe that research and evaluation are needed – especially in the education world. I am currently a second grade teacher and am always looking for new insights from research studies to bring into my classroom. Something I have been wondering since starting this course is if research and evaluation would have much effect without each other. Can we have research without evaluation? Can we have evaluation without research? I have an idea on where I stand for this, but would love to know how you stand!

    Thank you for your time,
    Kayla Boling

    1. Personally, I don’t think either is effective without the other. Researchers use evaluative tools to determine what is worthwhile to study, the best way to study the concepts of interest, and how to interpret the results; furthermore, research needs to be evaluated for its quality and other criteria (e.g., the field of research evaluation). Similarly, evaluation needs research to better inform both how we do our evaluations, such as through research on evaluation (RoE) and the program theory underlying what programs due (e.g., theory-driven evaluation). I’m not sure which would be more effective on its own.

  5. Thank you for sharing the interesting results of your study. Prior to beginning specific inquiry into the fields of research and evaluation, I was unaware of the debate and lack of consensus on the two topics. Quite frankly, I am a bit shocked at the collective effort that has been focused on finding new and unique ways of describing the relationship between the two concepts.

    You mention in your closing thoughts the possibility of evaluation moving towards existing as a separate profession. What do you believe needs to occur to move in that direction? Would there be any negative ramifications for either the research or evaluation field if evaluation did evolve into a unique and distinct discipline?

    1. There is a large literature on professionalization in evaluation which would help answer your first question. Negative ramifications might include putting boundaries on the profession that are not inclusive of people who currently do evaluation but do not fit the profession’s idea of what evaluation is or who evaluators are.

  6. Hello Dr. Wanzer,
    I am currently a graduate student at ASU. This article really helped me to clarify the difference between research and evaluation. As a visual learner, the diagram of the 5 ways research and evaluation could be related helped me to understand how parallel research and evaluation can be. I also enjoyed the link to LaVelle’s article. My question for you is do evaluators collaborate with researchers on their findings and how does that in turn affect researchers?

    Thank you!

    1. Yes, a prime example would be National Science Foundation (NSF) or Institude of Education Science (IES) grants which require researchers collaborate with evaluators. Outside of federal and state grants, I’m not sure of as many examples. Some people do both research and evaluation and are perhaps best situated to span the boundary between the two.

  7. Hello!

    I’m currently a graduate student at Arizona State University and this blog post was very interesting! The image displaying the multitude of ways that evaluation and research can overlap with each other was fascinating and very helpful. I was surprised that evaluators saw greater distinction between evaluation and research than researchers did! As someone who is studying Gifted Education, I can’t help but imagine how this applies to my field. I find that those who are evaluating gifted programs aren’t always aware of the research behind the programming, or even that those crafting the programming aren’t fully aware of the research! I wonder if part of this is that evaluators tend to think of evaluation and research as more separate than they perhaps should be.

    In my personal opinion, I think it makes sense for them to be continuous subsets of each other. Research influences evaluation and evaluation influences research. Both have the similar general aim to improve something. However, it often takes evaluators to put research and change into motion as they evaluate programming. The cycle continues as researchers continue to seek more answers as they observe the results of the programming. They are intertwined and I think it’s a good thing for all educators to be aware of! The process of researching, applying, evaluating, and then researching again is a very human thing.

    Thank you for a fantastic blog post!
    Liesel Lutz

    1. I completely agree with your sentiment that they should influence one another. In some ways they do, with research on evaluation, theory-driven evaluation, and when evaluators publish their findings for researchers to better find. I just wonder how we can better intertwine the two.

  8. Dear Dana,
    Thank you for posting a very detailed article on the differences between Research and Evaluation. As part of my graduate course, I have been reading on this topic, and would appreciate it if you could share your views on whether Research should be considered as a subset of Evaluation, or should it be considered vice versa? Do you think if both perspectives are worth considering? Thank you for your time. I look forward to your valuable thoughts.
    Kind regards,
    Ainee.

    1. I think ideally, if we can get to the alpha discipline Scriven describes, we would eventually get to a place in which research is considered a subset of evaluation. Currently, given the newness of the professional field and our lack of general recognition as a field, I don’t think we are there just yet.

  9. Dr. Wanzer,

    I enjoyed reading your blog post as it was quite informative and helped me understand the difference between evaluation and research. As a researcher on an early childhood project for Arizona State University, I admit, like your own research, my initial thoughts were that evaluation was a sub-component of research. I can say my team’s ultimate goal is to improve something, using research to identify and prove where and what needs improvement. So, after reading your blog post along with Mathison’s chapter, I see how research and evaluation differ, but also how the terms are interconnected.

    I’m curious, of Mathison’s five ways in which evaluation and research could be related, what do you believe is the most accurate description?

    Thank you,
    Joanne

    1. I think I would go with the majority of my fellow evaluators and vote for the overlapping Venn diagram, with an eye towards figuring out how we can get to the alpha discipline Scriven describes which would suggest research as a subset of evaluation.

  10. Hi Dr. Wanzer,
    My name is Emily and I am a graduate student at the University of Arizona. Like so many others, I found your article very helpful and thought provoking. I am fairly new to learning about the topic of research and evaluation and never knew there was such a distinction between the two until now. I grasp the idea that in general, the audience of researchers and evaluators differs. But what about social researches and evaluators? Is the purpose of both a social researcher and evaluator to ultimately meet the needs of various people and environments? I am interested to hear what your understanding of this is.
    Thank you for your informative post and the incredible work that you put into this topic.

    1. Given that I did the study with education researchers at AERA, I think it already answers for social researchers. I would be curious to compare to those in economics or in auditing because I think they might have very different conceptualizations of what evaluation is.

  11. Heather Keeley

    Hello Dr. Wanzer,

    My name is Heather Keeley and currently I am a graduate student. I am learning the difference between research and evaluation for the second class. We are now going more into the process, but what was strange for me was that before graduate school, I thought I knew the difference. What has happened is that I have gotten more confused. Confusion just means that my perspective is changing and I am being challenged!

    I wanted to first tell you how much this blog post has helped identify my own beliefs with research and evaluation. I looked at Mr. Lavelle’s blog post, and like him I am a very visual learning. The hourglass resonated with what I believe for research and evaluation based off personal experience and the textbook for our class “Research Methods in Education and Psychology: Integrating Diversity with Quantitative and Qualitative Approaches” by Donna Mertens. I believe that they overlap in similarity, but ultimately it is about the stakeholders involved. I am starting to see that there is more stake in evaluation than in research. I teach middle school, so how I am connecting this is to “bullying.” I was bullied, so I see that my own “merit,” “value,” and “worth” were based on those who were evaluating me. I listened to their “results” and changed to fit their definition, so that way I could be more accepted and effective at life. If they spent more time getting to know me and doing their research on me, the results would have changed, but if they just researched me with no evaluation, it would not have affected me. I would not have known their opinions on how to be better. It is not exactly the best metaphor, but it is how I am connecting to it personally.

    I have been researching myself during this quarantine and evaluating my selfworth based on old information and facts. It is time I evaluatd from a different perspective. When my professor asked which paradigm I think is important, I picked constructivist. I have to connect personally to what I am learning and create a narrative to learn. Connecting it to the job is just a little harder currently because my job is not about learning, but completion right now. The skills my students are learning right now are the ones not taught in the classroom.

    Thank you for taking the time to read my input on your blog post! Thank you for giving me an insight into this definition and helping with the clarification process!

  12. Dr. Wanzer,

    I am a graduate student in Learning Design and Technology at Arizona State University. Your article was very helpful in defining research vs evaluation. I initially thought that both terms were related. Upon reading your article and exploring the links provided, I think I have a better definition of what the two terms mean. I originally thought that they were the same because they both deal with identifying data and information. As an experienced evaluator, do you think that research and evaluation is necessary?

    Thanks again for your insight.

    Sincerely,
    Carl

    1. Hi Carl,

      I don’t think I’d be in this profession if I didn’t think that research and evaluation is necessary! How else do we know what works and for whom? How else do we expand the knowledge base? Research and evaluation help us in both endeavors.

      -Dana

  13. Dr. Wanzer,

    I am studying Curriculum and Instruction at Arizona State University and currently enrolled in a research course. I really appreciated your article and visuals you had to help explain the five ways research and evaluation could be related. I also appreciated the links to other pages that support your information and findings on research and evaluation. Thank you for the resources, such as Open Science Framework. Do you believe that evaluators are seeing the difference between the two more than researchers are?

    Thank you again!

    Lauren Foley

    1. Hi Lauren,

      Yes, my research found that evaluators see more differences between research and evaluation than researchers do. In other words, we see the uniqueness of our work more than researchers do.

      -Dana

      1. Dr. Wanzer,

        I completely agree that educators see more differences in evaluation and research. I teach fourth grade and I am currently attending graduate courses through ASU. We are studying the differences between evaluation and research, and the best way for me to break it down was to think how it applies to my classroom. As teachers, we are constantly evaluating our students through collection of artifacts. I use what I learn from evaluating my students to research ways to better help them meet their learning goals. Thank you for providing another source of information for my coursework and classroom!

        Sincerely,
        Megan Corder

  14. Paige Russell

    Dr. Wanzer,
    I’m enrolled in a graduate course at Arizona State University that is currently discussing the differences between research and evaluation. Upon searching for resources to inform my discussion post, I came across your article. Thank you for including helpful links, diagrams, and resources. I found it fascinating that evaluators are more likely to view evaluation and research as only intersecting whereas researchers leaned more towards evaluation as a sub-component of research. Before taking this course and learning about these differences, I honestly thought they were one and the same. Now that I know how evaluation goes beyond the analysis of data into determining value (worth and merit) of the findings, I feel as though I agree with the comment above mine as research and evaluation being cyclical in nature with both research and evaluation informing one another. I’m a classroom teacher and I have definitely had my students evaluate their findings but used the verbiage of research! Now that I can distinguish between the two, I want to further my own understanding on the differences and pass that along to my own students.
    Thank you again for your wonderful article- I found it to be incredibly helpful and straightforward.
    Sincerely,
    Paige

  15. Jennifer Baron

    Dr. Wanzer,

    I couldn’t agree more with your claim that many people new to evaluation struggle to understand it and explain the relationship it has to research. I am a middle school teacher pursuing a graduate degree in Learning Design and Technologies. It wasn’t until I started taking graduate courses that I even tried to tackle the concept of evaluations. As a teacher, and a pretty new one at that, I am used to assessing students and having administrators evaluate me, so I’ve never really done the evaluating myself. This makes it very difficult for me to fully understand what evaluations are, what they involve, and how they relate to research. However, after reading what you have to say about it, combined with the text I read for my course, I have a much better grasp of it.

    The graphic you included was very helpful when it came to my perception of evaluation relating to research. I can see how all five are plausible relationships and I think they can change depending on the content that is being researched and evaluated. Because of this, I tend to gravitate toward the idea that they exist on a continuum.

    Something I’ve been thinking about is whether or not research can be effective without an evaluation component and vice versa. I would love to hear your thoughts on this because I’m having a hard time deciding. I am inclined to think that research can be effective without an evaluation, but an evaluation cannot be effective without research. However, I don’t know if this is an accurate claim or if I’m considering all possibilities. I would appreciate your perspective on this idea.

    Thank you for your valuable input on this topic.

    Jennifer Baron

    1. Hi Jennifer,

      Great questions! I think they’re more synergistic that many believe. We see research being used in evaluation a lot, especially in theory-driven evaluation. However, I think we should also be thinking about research on evaluation to inform our evaluation practices.

      The incorporation of evaluation into research may be less obvious, especially with research tending to want to remain ‘value-neutral.’ Yet researchers embed their values into their research when they choose what and who to study, with what methods, with what data analysis tools, etc. (these might be the “merit” of a study). Approving a study for a grant or a paper for publication indicate the value we place on research (these might be the “worth” of a study). And statistical significance and effect sizes indicate the value we place on the findings (these might be the “significance” of a study). Research has evaluation all throughout it, which is why we need to embrace and advocate for the role of evaluation as a transdiscipline (and perhaps, as Scriven argues, the alpha discipline).

      -Dana

  16. Jackie Arthur

    Hi Dr. Wanzer,

    I am currently in a graduate program that focuses on learning design and technology, which involves me taking courses on both research and evaluation. I would definitely consider myself a novice when it comes to evaluation considering I have only taken one formal course on the topic and in my career life, I have evaluated learning programs, but not to the scope of a true evaluation.

    I would have to say that study you conducted and the topic mentioned is extremely common since I really do struggle to this day to explain the difference between research and evaluation. I know from my courses and experience that they’re not one in the same, but it does become difficult to explain the differences to others. I would say I connect with the participants that believe research and evaluation are similar in scope to a venn diagram. I have always felt that they overlap in some regard because there is a bit of research necessary in the evaluation process to either understand the program being evaluated or the company being evaluated. I believe that research can further lift up the evaluators in their endeavor to evaluate.

    I also will say that it is important to distinguish the difference between evaluation and research when working with a client or organization so that they understand the scope of exactly what you are there to do as an evaluator. Your tips are extremely helpful!

    Thanks for your insights and even the graphics help explain the different beliefs around this topic even further.

    Sincerely,
    Jackie Arthur

  17. Heather Allen

    Dr. Wanzer,
    I wanted to let you know how much I enjoyed this article. I’m a graduate student and between my recent class about evaluation and current class about research, this topic is coming up a lot for me. Until I took the course on evaluation last semester I thought evaluation and research were basically the same things. I know better now. This article added more to my knowledge in this area by pointing a lens at the evaluator’s perspective. I think it is a great perspective to have. Thank you for including the links to the data, diagrams, and other information of the study. I downloaded all of them and, as a beginner to the field of evaluation loved seeing the things I was taught in my class “come to life” in a real-life scenario. I thought the data in your figures and tables were easy to understand and the results are enlightening.

    As I am a novice in the field of evaluation, do you have any textbooks or other books you suggest to teach someone about evaluation? Any favorite resources you use (blogs, books, other publications) that you consider “must-read” information to stay up-to-date in the world of evaluation? I’ve already read the article from Patricia Rogers suggested in your post and will read the others you list above as well but I’m very interested in both Research and Evaluation and would appreciate any suggestions you might have from your experiences.

    Thank you!

    1. I find AEA365, #EvalTwitter, and following my favorite evaluation journals (e.g., American Journal of Evaluation, Canadian Journal of Program Evaluation, Evaluation and Program Planning, Journal of MultiDisciplinary Evaluation) the best way to stay up-to-date in the world of evaluation! There are a ton of resources out there, and the community is always sharing in a wide variety of mediums including blogs, podcasts, journal articles, webinars, and more.

  18. Pingback: #EvalTuesdayTip: Identifying relationships between evaluation & research - Khulisa

  19. Erika Stanischewski

    Hi Dana,

    Your post is of particular interest since it has prompted me to reflect on the relationship between evaluation and research within the context of teaching. I am a high school science teacher, and I have always regarded myself primarily as a deliverer of information that I hoped would become knowledge to the receiver. I did not consider myself an evaluator at the same time, and it was only until recently that my thought process began to change.

    When reading your post, I was immediately inclined to support the Venn diagram, and thus the hourglass model that LaVelle suggested in his post, to represent my idea of how evaluation and research are related. I can appreciate that research and evaluation both have their respective purposes, research being to discover “generalizable knowledge” and evaluation to discover “context-specific knowledge” (Alkin, M. C. & Taut, S. M., 2003), but that they share common methodologies and analysis techniques to render their conclusions.

    However, when I think of my role as a teacher, I see myself as using the results of an evaluation to illicit more research, to then test out and re-evaluate. I see myself learning through this process. When you say, “many evaluators also think evaluation is about learning, informing decision making, and improving programming, indicating the purpose of evaluation beyond simply the process”, it validates my belief that evaluation can elicit more research and can result in learning on behalf of the evaluator for the benefit of the stakeholders: my students.

    For example, I am evaluating whether a specific set of laboratory skills have been internalized by my students, and whether they are then transferrable to another scenario. Through the evaluation, I discover that the transfer of knowledge did not occur. Therefore, I research different methods for ensuring transferability of knowledge and then try the evaluation again to see whether my new approach has worked. Within my context of teaching, I am seeing evaluation and research as cyclical in that there really is no beginning or end; whether we begin with research that we then evaluate, or an evaluation that leads to more research. Either research or evaluation prompts the other in a cyclical manner. I wonder what your thoughts are in regards to my perspective.

    Thank you for a great thought provoking post, and congratulations on your work in the field of evaluation.

    1. I definitely agree that research and evaluation can and should be a cyclical process! I think we think of research informing evaluation more so than the other way around, but they can both improve how we think about the other. Research can inform us on what might be useful programs or interventions, but evaluations can elucidate things that might need further research to determine whether it can generalize to other people, situations, contexts, etc.

Leave a Reply to Paige Russell Cancel Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.