Hi! I’m Jonathan Scaccia, principal of the evaluation firm The Dawn Chorus Group and founder of PubTrawlr. What connects these two strands of work is finding better ways to efficiently and effectively extract meaning from text-based data. There is so much qualitative data out there that is laborious to analyze. Techniques from Natural Language Processing can help us to accelerate and scale the analyses.
Lessons Learned:
Sentiment Analysis. Words don’t just communicate ideas. They can express emotion. And, emotion is a critical aspect of how people relate to one another. Sentiment Analysis is an NLP technique that looks at the emotional content behind the words.
Sentiment analysis is well-established in customer service. It can be an essential technique to learn how customers perceive the value of a product. Although many research studies have explored the potential of these methods in behavioral health, they haven’t yet gotten a significant foothold in social-services evaluation.
For example, I looked at the overall sentiment communicated on Twitter by the two U.S. congressional representatives who represent my community: Rep. Dan Meuser and Rep. Chrissy Houlahan. Both were recently re-elected to their second term, so their official accounts are nearly the same age.
After pulling all the tweets using Twitter’s API, I used a pre-coded group of words called a lexicon to identify words as either positive or negative. For this, I used the bing lexicon. I then averaged the overall sentiment for each month and plotted these since the beginning of 2020, when they took office.
We can see that both generally use positive messages, though Rep. Houlahan communicates far more of them. Interestingly, we don’t see as much of a dip during the early days of the pandemic (April-ish 2020), which shows up in other accounts. Rep. Meuser has also been trending down since his party lost power in the November 2020 election.
There are other methods out there that can get at more nuanced emotions. For example, the NRC lexicon looks for words that signify emotions like Trust or Anticipation.
Rad Resource:
Text Mining in R. This open-source text by Julia Silge and David Robinson is a great entry-level way to explore implementing these methods in R.
It’s not that difficult to learn and can yield super interesting insights. Have fun with it!
The American Evaluation Association is celebrating Behavioral Health (BH) TIG Week with our colleagues in Behavioral Health Topical Interest Group. The contributions all this week to aea365 come from our BH TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Dear Jonathan Scaccia:
Thank you for sharing this text mining software, and the discussion as to how emotionally-coded language may be tracked statistically and henceforth analyzed. I agree that language, words, express emotion just as much as a semantic message. Greek Philosopher Aristotle articulated this in the 300’s (BC) when he introduced the concepts of ethos, logos, and pathos: specifically with regard to your example, he argued how a speaker can adjust his language to evoke an emotional response in the audience, known as pathos. This explains why we can be moved to tears reading a eulogy in the newspaper for someone unknown, or become enraged at the editorial on the other side of the page that regards something which we know nothing about.
You noted the distinction between positive and negative words. We know this as, perhaps, connotation: the general feelings associated with a term. While this is of course outside the scope of your fascinating blog post (and the purpose of the tool), it invites me to think about what exactly contributes to these positive and negative sentiments. We seem to observe two curves when you observe your data from 2019 to current. Some, yes, like you say, can relate to sociopolitical happenings. Though I am interested in your findings in that April 2020 period. I am curious, now, expanding this idea outward, about the extent to which time and place (i.e. culture, social context, political context, etc.) play into these findings. This study was conducted on Twitter from an American lens, I understand: I am curious if there’d be trends with nations that share similarities to the USA…
Thank you for sharing this fascinating response. As an English literature teacher, I am all too enthusiastic to see any interdisciplinary connections between artistic expression and the scientific method. I will be sharing this post with my colleagues, if you don’t mind!
Sincerely,
Kaitlin