Hi! My name is Shelita Hall-Augustus. This summer, I worked with the John D. and Catherine T. MacArthur Foundation through the AEA Graduate Education Diversity Internship (GEDI) to answer a question that affects all of us: can AI identify evaluation methods the way human evaluators do? I tested ChatGPT and Gemini against ten MacArthur Foundation evaluation reports. The work offered practical lessons about what AI can do well, where it falls short, and why human judgment remains essential.
Lessons Learned
Methods are more than searchable words
Both tools performed well when reports used straightforward language. When a report said “we conducted semi-structured interviews” or “we distributed a survey,” ChatGPT and Gemini recognized those methods consistently. The challenge came when evaluators embedded methods into narrative, describing how they tracked community sentiment or interpreted program data. The tools struggled to separate a data source from an actual analytical procedure. When writing becomes descriptive instead of clinical, AI gets confused. That’s when human evaluators come in.
The tool you choose shapes what you find
The two models process text in fundamentally different ways. ChatGPT stays literal, focusing on what is written on the page. Gemini attempts to read between the lines, looking for underlying themes. In one report, Gemini recognized that evaluators were using inductive reasoning based on surrounding context, but ChatGPT missed it because the word “inductive” never appeared. If you rely on one platform, you’re getting a particular kind of extraction, shaped by that tool’s personality and limitations.
Hot Tips
Upload strategy changes everything
The most important finding came from something I didn’t expect. When I uploaded all ten reports to ChatGPT at once, it gave me 95 candidate methods. Using those same reports and the same prompt but uploading them one by one, ChatGPT generated 862 rows. Gemini went from 22 rows to 121. You might think more rows means better work, but it doesn’t. When AI focuses on a single document, it becomes overzealous, treating simple program metrics as research methods and splitting one survey into twenty rows based on individual questions. Those extra rows may be false positives and fragmented data. More volume may just mean more cleaning.
Prompting is methodology
My original prompt was too broad, asking for a structured table showing what methods were most used. That gave ChatGPT permission to summarize, placing interviews and focus groups in the same cell. The revised prompt restructured the approach. The AI took on the role of an expert evaluator and analyst. Each individual method received its own row. Exact page numbers became non-negotiable for every claim, which helped catch hallucinations. The AI also had to explain how each method was executed, not just name it. How you structure your prompt shapes what data comes back and how much verification work you’ll need to do.
Rad Resources
Mainstream models miss nuance because they’re trained on dominant, Western datasets. Tools like Aisha, Latimer, and Cohere’s Aya recognize linguistic patterns and lived experiences that standard AI overlooks. Running one alongside your workflow offers a second lens on community context.
AI works as a strong first-pass tool. It moves through hundreds of pages quickly and flags where methods might be hiding. What happens next is where human judgment takes over. You open the document, go to the page the AI cited, verify what it found, and separate procedures from data sources. Because the goal isn’t just to move faster or produce bigger spreadsheets. The goal is to make sure the people and communities behind the data are still seen and understood correctly.
The American Evaluation Association is hosting Graduate Education Diversity Internship (GEDI) Program week. The contributions all this week to AEA365 come from AEA’s GEDI Program and its interns. For more information on GEDI, see their webpage here: http://www.eval.org/GEDI Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.