Hello! My name is Dr. Kinsey Simone, and I am an instructor of graduate Quantitative Research Methods at Tennessee Tech University in Cookeville, Tennessee. My PhD is in Program Planning & Evaluation, and my primary research and evaluation interests pertain to the effectiveness of large-scale events to decrease stigmas towards and increase awareness of invisible disorders, like OCD (obsessive-compulsive disorder). When I was first introduced to sentiment analysis in 2023, I was excited to use a new and efficient tool for qualitative survey responses. However, I recently found that sentiment analysis may not be the best approach for all projects/evaluations. This post shares important considerations for evaluators and researchers regarding when (and when not) to use sentiment analysis, based on my experiences.
Why is this important?
As evaluators, we use a variety of tools and measures to assess programs or projects. Surveys can allow us to evaluate stakeholder perceptions, but qualitative responses can be time-consuming to analyze using traditional methods. Sentiment analysis provides an efficient, objective way to analyze qualitative responses, but in certain contexts, its limitations can outweigh its benefits.
What is sentiment analysis, and how can it benefit me?
Sentiment analysis is a computation program which uses lexical methods to sort text through identifying emotions, or sentiments (e.g., positive vs. negative, happy vs. sad, etc.). It can reduce subjectivities or biases because it is purely computational and can be used for survey responses, written reflections, social media posts, transcripts, and more. For more on using sentiment analysis and R Software coding programs for this tool, visit here.
Sentiment analysis sounds great! What’s the problem?
It is important to remember that computers cannot understand social contexts and do make mistakes. Reporting incorrect sentiments can have detrimental effects, especially in contexts where misinterpretations of results can have dire consequences for an evaluand/project. For example, I recently conducted a study to assess pre-service teachers’ (PSTs’) perceptions of and stigmas towards OCD in education and in general. PSTs responded to three qualitative questions regarding their perceptions of OCD, which I analyzed using sentiment analysis. One high-frequency positive sentiment was “perfect,” but I soon realized that PSTs only used this term in stigmatized, negative descriptions of a student with OCD (see Figure 1). Because OCD is a sensitive topic, I also realized that the computer misinterpreted many of my participants’ words as the wrong sentiment. I had to trash all sentiment analysis results!
Example of Misinterpreted Sentiment Analysis Outputs
Lessons Learned
Based on the above experience and similar ones over the past year, I provide some “do’s” and “don’ts” regarding when and when not to use sentiment analysis below.
DO use sentiment analysis if:
- Qualitative data all pertain to one primary focus or question.
- You have a large dataset.
- Responses are not sensitive in nature/cannot be misconstrued.
- You are close to the evaluand/project and/or stakeholders/participants and want to cross-reference with objective results.
Do NOT use sentiment analysis if:
- Qualitative data do not all pertain to one primary focus or question.
- You have a small dataset.
- Responses are sensitive in nature/can be misconstrued.
- You do not have time to cross-reference high word frequencies with raw data.
Sentiment analysis is a great tool for analyzing qualitative data quickly and without bias. However, like all analyses, it comes with important considerations prior to use. This blog provides a brief overview of why context is important to consider when deciding whether to use sentiment analysis for evaluators and researchers.
The American Evaluation Association is hosting PreK-12 Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our PreK-12 Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.