Using Text Analysis to Identify New Areas of Improvement in Evaluation by Sy Islam, Michael Chetta, and Andrzej Kozikowski

Hi, we are Sy Islam, Michael Chetta, and Andrzej Kozikowski from Talent Metrics, a data driven consulting firm. We’re writing about using text analysis to identify new areas of improvement when conducting an evaluation. Evaluation research often uses surveys or other forms of data collection that limit potential outcomes of the evaluation. When designing a survey, you may not cover all the areas of potential improvement using fixed response scale items. Text analysis is a phenomenological approach that allows research participants to provide more detailed responses, typically to open-ended questions. Rather than constraining responses that you think participants will give, text analysis and open-ended questions allow research participants to provide their own ideas and insights into the workplace. Qualitative approaches can be used to gain insight on quantitative results. Text analysis is an efficient way to analyze large amounts of qualitative data rather than traditional methods that require more time and resources.

Hot Tip: Use open-ended questions, social media comments, and text analysis to look for areas of improvement and areas of weakness in an intervention or program.

When evaluating a program or site, open-ended items should be included in any evaluative survey. Another great source for open ended comments would be social media posts and pages. Every major business or service has a Facebook, Yelp, or Google profile. These resources can provide rich and surprising data.

We conducted an evaluation of urgent care centers to identify what language drove positive as well as negative ratings.  Organizing comments from Yelp into positive (4 or 5) and negative (3 and below), we identified language that was used most often in those ratings.  We had hypothesized that different language drove urgent care ratings on social media, and that they fell into three broad categories: 1) Facility, 2) Care Delivery and 3) Staff. We were surprised that patient reviews, both positive and negative, were driven by the same types of experiences.

Top words found in positive and negative reviews

Rad Resource: We used Tropes 8.4 to analyze the text from the social media reviews. Tropes is an open source software used in academic and applied research. It can also provide a graph of actors like the one below that illustrates the relationships between review quality and the language that is used.

Graph of Actors: Relationship of "negative reviews" to patient evaluation terms of interest

Lessons Learned: Evaluation of businesses, nonprofits, and their associated programs are often conducted using surveys or other forms of quantitative data collection. These methods often include the evaluators’ own assumptions about what is important in driving key outcomes. By using qualitative data, such as social media comments, new and unexpected discoveries can be made. Remember to listen to program participants or those who utilize your services. They are the real subject matter experts and can help you as you evaluate.

Rad Resources:

Two  text analysis articles we wrote that go into further detail regarding our projects.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.