Hi, I am Steve Powell, writing for the International Evaluation Academy. As an evaluator I have mostly worked in international development; now I spend my time at Causal Map Ltd in the UK.
Evaluation is increasingly asked to support transformational change across economic, social and environmental systems. Much of that work depends ultimately on the analysis of text: interview transcripts, project reports and open survey answers. AI now offers to do that analysis for us, and the temptation is to hand it the whole job at once, for example “summarise these interviews” or “what are the main outcomes in this report?”. Some evaluators are doing just that, and others are exploring more careful ways to use the technology.
The trouble with big questions like “summarise this document”, “what matters most here?” or “what are the main themes?” is that they are already evaluative judgements. Ask an AI to make them and it answers in one opaque step, with no working you can inspect and a real risk of a confident but wrong reply. When the findings may inform change across whole systems, that opacity is a problem.
One alternative is to break the work down. Split the big question into many small, precise tasks that careful people would mostly agree on the answer to. Instead of “is this programme working?”, ask of each paragraph: “does this mention a change in health behaviour?”, and then “is that change described as positive by the speaker?”. Each small task has an answer you can check, and the pattern of small answers is what lets you understand the system as a whole.
Hot Tips:
- Only give the AI tasks whose answers you could verify yourself on a sample of data. If you cannot check it, do not trust it.
- If you give an AI a “big black box” question, it may include accurate quotes in its answer, but making an untraceable judgement and then finding a matching quote is not yet rigorous text analysis.
- Keep the AI on low-level work: spotting, classifying, extracting. Keep the evaluative judgement, and the job of reassembling the small answers into a finding, for you and your team.
- Write the steps and instructions down, including any rubrics, so another evaluator could in principle follow them by hand.
- Small, well-defined tasks can often run on smaller, less energy-hungry models rather than the largest frontier model, which can help keep the environmental cost of your analysis down.
Lesson Learned:
This issue is not specific to AI. Most text analysis breaks text into small coded units, then builds findings back up. AI does not replace that discipline. It lets you apply it to far more text than you could by hand, consistently and at low cost. The responsibility for how to break the question down, and how to put the answers back together, stays with you.
Rad Resources:
I have written here about how we use this kind of approach when doing causal mapping for evaluation.
Do you have questions or experiences to add? Please use the comments on the aea365 page.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please do so using the form on our website. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Well done, Steve Powell! This is well written and very helpful. I like the idea of small tasks which we can use human intelligence to track back, and then the element of replicability of our methodology. This is Great!
I’d love to hear what we all think the advantages of using AI for tasks like this are.
If, to trust AI, you need to put work into breaking a task down into sections that you as a human can verify because you can’t trust AI to do the work as well as a human can, maybe the issue is you SHOULD be using a human you trust.