Artificial Intelligence in Evaluation: Transcription with Otter by Tatiana Masters

How can an adorable marine mammal help with my qualitative evaluation work, you ask?

I’m Tatiana Masters with Evaluation Specialists, and I’m going to tell you. At Evaluation Specialists, we work with non-profit, philanthropic, and government clients to help them evaluate and improve their education, prevention, health promotion, and social service programming. Because lots of our projects involve qualitative interviews and we are always looking for ways to work more efficiently, we recently decided to try something new: Using a software product called Otter to do the transcribing for us.

For years I’d wondered about speech-to-text technology, also called artificial intelligence (AI) or machine transcription. Some qualitative colleagues had reported being disappointed in its performance, but I knew the technology was maturing, and I’d heard good things about Otter… including that the basic version was free.

How does the transcription happen?

The software records speech, or you can import a digital recording you’ve made elsewhere. Then it uses voice recognition algorithms to produce a written transcript. It synchronizes sound with text so that you can click a word in the online transcript to hear what was said on the recording. This lets you edit the transcript online before you download it for analysis.

What’s the quality like?

Otter produces a decent, mostly verbatim transcript that includes handy time stamps. The punctuation can be a bit odd, and there are occasional but notable inaccuracies that you’ll want to correct by hand. For example, I recorded a conversation with a colleague as a test. Although that transcription was good overall, I had to change “touchdown is an honor” to what she’d actually asked me, which was “Tatiana, is it Otter?” However, even the best professional transcription isn’t completely accurate, and I’d definitely use Otter again.

Lessons Learned:

1. Otter produces a pretty good verbatim transcript at a very low cost.
2. Everything happens online, so a good internet connection is essential.
3. It’s most accurate when transcribing 1:1 interviews, but can differentiate multiple speakers if you identify or “tag” them, so might work for focus groups too. Email and let me know if you’ve tried this!
4. Otter works best in evaluations where the person who did the interview also analyzes the transcript. If this can’t happen, the interviewer will need to spend time listening to the recording and correcting the transcript online before downloading it. (Even accounting for this staff time, our total costs were still less than professional transcription.)

Hot Tip: Let your evaluation aims and available resources drive your decision about whether to use Otter.  For example, if you have a small evaluation budget and need to move quickly, it’s a great tool.

Rad Resource: Try Otter for yourself: https://otter.ai/

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.