AEA365 | A Tip-a-Day by and for Evaluators

TAG | qualitative analysis

My name is Teresa McCoy and I work as an Assistant Director of University of Maryland Extension (UME) with responsibility for evaluation and assessment. Like many Extension colleagues, my evaluation department consisted of me until last year when I was able to hire an additional person. The two of us have responsibility across all program areas in our organization, and I am always looking for technologies that can help save time while adding quality to evaluation efforts.

This past year, I needed to conduct about 25 interviews. I was faced with hours of conversation that would have to be transcribed and analyzed without assistants to help.

Several people suggested, given that I’m at a university, to “Just hire an undergraduate student. It shouldn’t cost you very much that way.” Well, I don’t know about you, but if I have spent countless hours preparing questions, designing a protocol, and contacting and scheduling interviews, I am not about to hand over transcription duties to the first student “off the street.”

Football solved my problem. I know that’s hard to believe, but while I was at a Baltimore Ravens football game party with friends, I was chatting with an education policy analyst. She told me about TranscribeMe!™ and her good experiences with the company and the product.

Lesson Learned: Hot tips and rad resources often are found at unlikely places!

Lesson Learned: After some investigation, I found out that TranscribeMe!™ and NVivo™ have a business partnership. I was able to upload my audio recordings from within NVivo™ (after setting up my account) and the transcripts were then sent back to me and into my NVivo™ project file. In the media options, there is a “purchase transcript” option.

purchase transcript and check status buttons

To clarify, you can use TranscribeMe!™ without having to use NVivo™. However, given that I was using NVivo™ for my coding, these two products made the initial work a lot easier and faster. I received some of the transcripts within 24 hours and almost all of them within 48. The transcript quality was excellent. And, as I am sure you’re wondering, the price was good (Price is negotiable depending on quantity of work, number of speakers, and other options.)

You can use the app on your smart phone to record. No special equipment needed.

Rad Resource: TranscribeMe!™ and NVivo™ partnership. TranscribeMe!™ at www.transcribeme.com

Rad Resource: Information from QSR about using NVivo and TranscribeMe!http://www.qsrinternational.com/products_nvivo_transcription-services.aspx

If you’re a football fan like me, now you have a great excuse to watch the games because you never know when you’ll find your new evaluation rad resource!

The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hi folks! I’m JT Taylor, Director of Research and Evaluation at Learning for Action (LFA), and I’m here with my colleague Emily Drake, Senior Consultant and Director of Portfolio Alignment. LFA is a San Francisco-based firm that enhances the impact and sustainability of social sector organizations through evaluation, research, strategy development, and capacity-building services. Emily and I can’t wait to share an easy and reliable approach to facilitating participatory, collaborative qualitative analysis processes at this year’s AEA conference.

Lessons Learned: Effective facilitation is essential for leading participatory and collaborative evaluation processes: (1) it helps us to surface and integrate a multitude of perspectives on whether, how, and to what extent a program is working for its intended beneficiaries; (2) it is necessary for building and maintaining trust among stakeholders: trust that they are being heard, that their perspectives are weighted equally among others, and that their participation in the evaluation process is authentic and not tokenized; and (3) it is important for producing the buy-in of stakeholders and relevance of results that ensure evaluation findings will inform real action.

Engaging a variety of stakeholders, including program beneficiaries, in the analysis and interpretation of data in a way that authentically includes their perspective and contributions is important—and takes a set of facilitative skills and tools that go beyond evaluators’ typical training in technical analysis. In our work implementing collaborative evaluations, we have found that the same facilitation techniques that produce great meetings and brainstorming sessions can also be used to elicit great insights and findings from a participatory qualitative analysis process.

Hot Tip: Use participatory analysis techniques when you want to synthesize qualitative data from multiple perspectives and/or data collectors—whether those data collectors are part of your internal team, evaluation partners, or members of the community your work involves.

  • Do the work of “meaning-making” together, so that everyone is in the room to clarify observations and themes, articulate important nuances, and offer interpretation.
  • Use a 1-2 hour working meeting with all data collectors to summarize themes and pull out key insights together. Have each participant write observations from their own data collection, each on a large sticky note. Then group all observations by theme on the wall, having participants clarify or re-organize as needed.
  • Save reporting time later by asking participants to annotate their sticky note observations with references to specific interviews, transcript page numbers, and even quotes from their data collection to make it easy to integrate examples and quotes into your report.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from th eAmerican Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Rebecca? They’ll be presenting as part of the Evaluation 2014 Conference Program, October 15-18 in Denver, Colorado.

·

Shirah Hecht here again, with a second method to move qualitative data efficiently from analysis to presentation using common desktop software.

Outline mode in Microsoft Word allows you to place text under headers and sub-headers that can be expanded, collapsed, and re-ordered – taking all of the text, when you move a header.  With this tool, you get an overview of a larger amount of text, and also see details as needed.

Learn about Outline mode in Word here.

Here’s the basic method to use this tool for qualitative data:

(1)   Open two documents simultaneously:

  1. Your original typed notes in Word (raw data file)
  2. A new Word document where you will work in Outline mode (analysis file)

(2)   As you read through your notes, create headers in your analysis file reflecting themes or topics.  Create and revise headers and sub-headers as needed, until you finalize them.

(3)   Copy chunks from your original text (paragraphs, extended quotes) into the analysis file, under the appropriate topical header in “normal” text with interview source.

(4)   As you copy, collapse and expand the outline to find relevant sections.

(5)   With the new nearly-raw copy of your notes, you can:

  1. Re-order individual quotes in a section to support analysis.
  2. See and review all quotes on a given topic, to write up your interpretation.
  3. Delete unneeded quotes; keep and clean the few you’ll use to illustrate.

This Outline Mode visual is from a final report; categories reflect completed analysis.

Hecht rev 3

Lessons Learned:

  • Use clear headers to revise the outline order easily.  Don’t be word-shy at first.
  • Expect three stages of the analysis document: all raw text copied into headers; draft report with narrative; final report with all the trimmings and sources noted or deleted.

Hot Tips:

  • As you code the text chunk, don’t think too much about the content, beyond the basic topic it addresses. Later, you’ll consider the results in light of all comments on that topic.
  • Everything that’s reasonably on the topic among the responses gets a code!
  • Highlight interview text in your original file, as you copy it to the analysis file to track your work. It’s also a great motivator to feeling you’re making progress!
  • Put your original interview or observation notes in a different color font.  Once you’ve completed your report, you can convert all text to normal font.

Rad Resources:

Great guides for inductively analyzing and reporting on qualitative data:

H. S. Becker, Tricks of the Trade: How to Think about Your Research While Doing It

H.S. Becker, with a chapter by Pamela Richards, Writing for Social Scientists: How to Start and Finish Your Thesis, Book, or Article

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Shirah Hecht and am an independent researcher and non-profit program evaluator.  Qualitative data can be overwhelming, and challenging to reach valid conclusions.  Here is the first of two methods to move qualitative data efficiently from analysis to presentation using common desktop software.

Excel’s Pivot tables lets you flexibly create subgroups and report standard quantitative summary measures on them.  Learn about Excel Pivot Tables here. Use the simplest subgroup descriptors – count and percent – to categorize and summarize qualitative responses (i.e., narrative comments) in two steps.

STEP ONE.  Create the dataset.  Enter each comment into a cell, with “Comment” or the prompt-question as the header, and attach a unique ID to each.  Then, reading each comment, develop and enter codes for meaningful themes.  Each theme header is a “variable” for this new dataset.

The table shows sample results of this step: each response is in a cell in one column, with ID to the left.  Each column header to the right names a theme identified, with codes you enter below.

Hecht rev 1

STEP TWO.  Analyze the results. “Pivot” the coded results, to find the number and percentage of comments in each thematic category.

Here’s a basic “recipe” for Pivoting the coded results.

  • Place ID in the “Values” box (summary data) and use “Value Field Settings” to select COUNT.
  • Enter ID again as COUNT.  Then use “Show Values As” tab to select PERCENT IN COLUMN TOTAL.
  • Place the theme you’d like to trace (e.g. “Theme1”) in the “Row Labels” box.
  • Option: Place the comment header (“Comment”) in the “Row Labels” box also.

The resulting table shows the count and percent of responses in the different theme categories.

  • Double-click on a COUNT: a new window will show all those comments.
  • Re-code comments to clean categories conceptually; then refresh table.
  • Re-group, re-order and re-label categories for analysis and reporting.
  • Copy and paste comment lists or a table into your report.

This Pivot Table visual shows how you would see the three comments given above.

Hecht rev 2

Lessons Learned:

  • Copy focus group comments easily into Excel by typing in paragraphs: person, colon, comment.
  • “Power through” coding.  Keep concepts clear: does it go in this category or another one?

Hot Tips:

  • Use one spreadsheet for each open-ended question.
  • For multiple codes, duplicate the comment with a new ID.
  • Create ID numbers for focus group comments, in discussion order.

Rad Resources: Pivot tables uses OLAP technology, which goes beyond the standard crosstab table.  You can learn more about the connection between the two here.

My favorite text for writing surveys, which include open-ended questions:  Seymour Sudman and Norman S. Bradburn, Asking Questions: A Practical Guide to Questionnaire Design.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Archives

To top