Hi, we are Sy Islam, Michael Chetta, and Andrzej Kozikowski from Talent Metrics, a data driven consulting firm. We’re writing about using text analysis to identify new areas of improvement when conducting an evaluation. Evaluation research often uses surveys or other forms of data collection that limit potential outcomes of the evaluation. When designing a survey, you may not cover all the areas of potential improvement using fixed response scale items. Text analysis is a phenomenological approach that allows research participants to provide more detailed responses, typically to open-ended questions. Rather than constraining responses that you think participants will give, text analysis and open-ended questions allow research participants to provide their own ideas and insights into the workplace. Qualitative approaches can be used to gain insight on quantitative results. Text analysis is an efficient way to analyze large amounts of qualitative data rather than traditional methods that require more time and resources.
Hot Tip: Use open-ended questions, social media comments, and text analysis to look for areas of improvement and areas of weakness in an intervention or program.
When evaluating a program or site, open-ended items should be included in any evaluative survey. Another great source for open ended comments would be social media posts and pages. Every major business or service has a Facebook, Yelp, or Google profile. These resources can provide rich and surprising data.
We conducted an evaluation of urgent care centers to identify what language drove positive as well as negative ratings. Organizing comments from Yelp into positive (4 or 5) and negative (3 and below), we identified language that was used most often in those ratings. We had hypothesized that different language drove urgent care ratings on social media, and that they fell into three broad categories: 1) Facility, 2) Care Delivery and 3) Staff. We were surprised that patient reviews, both positive and negative, were driven by the same types of experiences.
Rad Resource: We used Tropes 8.4 to analyze the text from the social media reviews. Tropes is an open source software used in academic and applied research. It can also provide a graph of actors like the one below that illustrates the relationships between review quality and the language that is used.
Lessons Learned: Evaluation of businesses, nonprofits, and their associated programs are often conducted using surveys or other forms of quantitative data collection. These methods often include the evaluators’ own assumptions about what is important in driving key outcomes. By using qualitative data, such as social media comments, new and unexpected discoveries can be made. Remember to listen to program participants or those who utilize your services. They are the real subject matter experts and can help you as you evaluate.
Rad Resources:
- Data Science Central Intro to Text Analysis
- An Introduction to Text Mining: Research Design, Data Collection, and Analysis. This is a good book to get started with the thought process behind text mining and text analysis
Two text analysis articles we wrote that go into further detail regarding our projects.
- A Qualitative Model for Customer Behavioral Decisions and Satisfaction in the Hospitality Industry
- Speaking of Urgent Care Centers: Language Matters
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.