Innovative #Eval Week: Schwalbe, Miller, and Gensinger on #EvalAHA! What Can Twitter Tell Us about the AEA Conference?

Hi, we’re Anne Schwalbe, Keith Miller, and Michelle Gensinger, graduate students in the Evaluation Studies Program at the University of Minnesota.

We chose Twitter as a platform for exploring creative evaluation methods at the AEA Annual Conference. These are the questions we hope to answer with our analysis:

• How was Twitter used during the conference?
• Which sessions at the conference generated Tweets?
• What did Twitter use look like during the conference (descriptive analytics)?

We had a lot to learn about Twitter and conference evaluation. We hope that our experience helps you decide whether Twitter can contribute to your evaluation practice.

Hot Tip – Be Prepared to Promote Twitter Use: Not all AEA attendees use Twitter. We wanted to increase twitter use for this project.

• Fun incentives can promote use without making anyone feel pressured.
• Have instructions about signing up for Twitter readily available.
• Demonstrate how Twitter can be used. A projector at our information table displayed AEA’s @aeaweb Twitter feed.

Hot Tip – Know Your Hashtags (#): Hashtags are essential to analyzing data. They “tag” all the Tweets related to a topic.

• Hashtags may already exist for an event, BUT you may need to create and promote them. AEA established #eval12 for the conference. We introduced #evalAHA to track learning moments inspired by the conference.
• A trusted twitter voice, @aeaweb, spread the word about our hashtag.

Hot Tip – Know the Limitations of Twitter Data

• The sample of individuals Tweeting was small and self-selected.
• Tweets are public.
• The format (140 characters) is restrictive.
• Hashtags don’t always work perfectly. Tweets labeled #eval2012 or #aea12 didn’t show up in our initial searches.

Hot Tip – Have an Analysis Plan

• Know your time frame and pull data as soon as possible. It gets tricky to grab Twitter data more than a week old.
• Manual coding of Tweets is time intensive; we’ve spent over 25 hours coding 1,500 Tweets to explore Twitter use at the conference.
• Free, quick, and easy online tools provide simple descriptive analytics about a hashtag or Twitter account.

Rad Resources

• Topsy.com: an online tool that instantly displays a summary of activity, top Tweets, and other fun stuff from any hashtag.
• NCapture: an NVivo 10 add-on that captures data from social media websites and does some auto-coding.
• Visual.ly: easy infographics templates for hashtags or twitter users. Here’s one for #evalAHA

We’re learning all this week from the University of Minnesota Innovative Evaluation Team from Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.