Greetings, fellow evaluators. Claudia Lopes, Affiliated Lecturer at University of Cambridge and Shravanti Reddy, Evaluation Specialist UN Women Independent Evaluation Service here to share our learnings from the recently published report ‘Can Big Data be Used in Evaluation?: A UN Women Feasibility Study’.
The quick answer… yes, but we still need to learn more and test how to incorporate big data as part of evaluation methodologies. 
Our task was to investigate the feasibility of leveraging big data sources – particularly Twitter, Facebook and radio data – to improve the evaluation of gender equality and women’s empowerment initiatives.
Here is what we tried to do…
- Develop and test a measurement model to select the best big data indicators for UN Women Twitter campaigns (Mexico) and Facebook posts (Pakistan).
- Identify important population biases, in terms of demographics and language.
- Analyse across geographies and over time to derive evaluation insights, disaggregating by gender.
- Triangulate results from big data sources with traditional qualitative methods.
- Discuss the limitations of this analysis such as selection of meaningful indicators and under-representation of certain groups.
Here is some of what we learned….
- Getting historical data from Twitter is time-consuming.
- The dominance of certain Twitter hashtags obscured others also relevant.
- Longitudinal analyses of hashtags, given their short life, was not meaningful.
- Most of the tweets analysed were retweets (about 75%) with the intention of the user unclear (e.g. agreeing, sarcastic, etc.).
- Crowd-coding and thematic analysis have proven more valuable than automatic sentiment analysis to code opinions.
- Many Facebook pages from organizations contain limited discussions and may have biased samples.
- Radio is an important social venue that can provide highly relevant and rich data, but requires careful recording and coordination.
Hot Tips: Are you planning to use big data methods in your next evaluation? Here are four things that we recommend:
- Understand the bigger picture of the social platform in a country before considering it as a data source for evaluation.
- Big data should be incorporated in the design of the evaluation from the outset and enough time should be allocated to request data access and build analytical models.
- Big data should precede traditional data when sequencing and evaluating. During the scoping stage, big data may reveal surprising case studies.
- Big data can be shaped in ways that enhance its value by promoting certain hashtags or designing radio programmes to gather audience data.
Finally, if you missed our UN Women presentation, join us for Is it possible to use big data for the evaluation of social development programs? at the AEA Conference in Cleveland on Friday, November 2.
Rad Resources:
Can big data be used for evaluation? A UN Women feasibility study (2018)
UN Women’s Gender equality and big data: Making gender data visible (2018)
Hot Tip: Stay Connected! Update your TIG selections here.
The American Evaluation Association is celebrating International and Cross-Cultural (ICCE) TIG Week with our colleagues in the International and Cross-Cultural Topical Interest Group. The contributions all this week to aea365 come from our ICCE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.