Greetings from Charlotte, fellow evaluators. I’m Faye Shaer, a Senior Evaluation Associate at Epiphany Community Services, an evaluation firm helping coalitions, nonprofits, and public-sector partners tell stories of impact.
In November 2024, we convened an AI workgroup at a moment when AI was moving faster than most organizations could keep up with. Staff were curious, cautious, and imagining how AI might support evaluation work, but the harder question was unavoidable: How should AI enter our operations, and where should we draw the line?
I have had the opportunity to lead this workgroup, and what began as a cautious exploration of emerging tools quickly became something broader: governance, staff training, platform testing, prompt library creation, and workflow-improvement effort.
More than a year and a half in, three lessons are clear:
Lesson 1: Start with rules, not tools.
One of the most important decisions we made was to slow down before jumping into platforms. Before we asked, “Which tool should we use?” we asked, “How can AI help us do our work better and faster while still protecting the standards that make evaluation credible?”
That led us to define approved and unapproved AI use cases. We identified lower-risk uses, including administrative support, thought partnership, and secondary data research. We also named boundaries around identifiable client data, sensitive internal information, and any output staff do not fully review. That early governance work gave us a shared language. It helped us define risks before evaluating platforms, and clarify what we needed a tool to do. By the time we moved into platform research, we were not chasing advanced AI features and impressive demos. We were testing for fit.
Lesson 2: Test where AI actually adds value in your workflows
Good governance builds confidence. Staff are more willing to experiment when they understand what is allowed, and where human judgment matters. Each workgroup member piloted using real-but-anonymized examples from our own work. That mattered because polished demos hide the messy parts of evaluation: unclear survey responses, inconsistent data exports, missing information, overlapping themes, and the need for careful interpretation.
We saw where tools saved time, but we also saw where they misunderstood context, overstated findings, or required stronger human review. For us, responsible AI use has meant protecting confidentiality, reviewing every AI-supported output, and disclosing AI use in client-facing deliverables. Our final platform selection was grounded in shared criteria, staff experience, and evaluation-specific needs.
Lesson 3: Your position on AI will evolve, and that is okay.
Today, our practice is nuanced. Staff need examples that connect directly to their work. We now support structured qualitative analyses, maintain a prompt library organized by tasks, and pilot agentic automations for recurring work like survey summarization, report generation, and others.
This shift happened because our understanding of the tools changed, our internal capacity grew through quarterly trainings and monthly updates, and our boundaries flexed accordingly. Static guidelines age quickly; living guidelines, revisited as the team’s knowledge evolves, hold up better.
We are still working through hard questions around client communication, confidentiality scrubbing, equitable access, and disclosing AI use without diminishing the value of human expertise. For our team, the answer has been to build the internal structures that allow us to use AI thoughtfully without replacing human creativity and output.
If your organization is building an AI workgroup or simply trying to make sense of where to begin, I would love to hear what questions you are sitting with. I am also looking forward to continuing this conversation at Evaluation26 during my roundtable session, “Building an AI Workgroup in an Evaluation Firm.”
The American Evaluation Association is hosting North Carolina (NC) Evaluators Affiliate Week. The contributions all this week to aea365 come from NC Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.