Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hello! I’m David Fetterman, PhD, a past president of the American Evaluation Association, founder of Empowerment Evaluation, and co-chair of its Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group. And I’m Paul St. Roseman, Ed.D., Chief Research and Information Officer at Reach University, the founder of Data Use Consulting Group, and a a former Stanford University student of David Fetterman. Together, we serve as critical friends, facilitating sessions with a focus on evaluation thinking and integrating AI into their evaluation systems. The focus of this contribution is on how we are helping to integrate AI into The Just Trust’s evaluation thinking and processes.
Introducing Gestalt Empowerment Evaluation
Gestalt Empowerment Evaluation is designed to help program staff and community members see the whole picture—particularly when essential elements are hidden, fragmented, or difficult to interpret. Gestalt theory emphasizes that the whole is greater than its parts.
AI tools enhance Gestalt Empowerment Evaluation because it excels at identifying patterns of proximity, similarity, continuity, and anomaly across large volumes of information—tasks that can overwhelm program staff and community members working under time and resource constraints.
Instead of following one of the popular 3-step or a 10-step approaches, Gestalt Empowerment Evaluation is embedded in the group, community, or organization. It is designed to be holistic and follow the natural flow of the community or organization’s pattern of events and life cycle.
Criminal Justice Reform Initiative
The Just Trust is an intermediary funding entity focused on reforming the criminal justice system. It is a $350 million Chen-Zuckerberg-funded initiative. The Chief of Staff and cross-functional teams are conducting the empowerment evaluation. They are immersing themselves in data collection, analysis, and reporting. They are taking charge of their self-evaluation organically, as issues surface and the organization pivots to meet its goals.
AI- Driven Empowerment Evaluation
Personalized GPT. We have helped The Just Trust develop a personalized GPT. Their data was uploaded to their GPT. Then they queried the “database” to summarize and make sense of their voluminous data. One of the cabinet leaders exclaimed in one of the sessions, “This is great, who would have guessed?” Another exclaimed, “It is answering my questions.” A third said: “It is making me ask questions I would never have thought to ask.” The ability to make sense of mountains of their data in seconds was an eye-opener.
NotebookLM. We also helped them create an Evaluation Repository. We used Google’s Notebook LM to create the repository. We curated the repository by uploading some of the most user-friendly, accurate, and useful evaluation materials. Documents ranged from PDFs to URLs. Topics included logic models; theories of change; empowerment, participatory, and collaborative evaluation approaches; culturally responsive evaluation; interviewing, surveying, and focus group guides; and qualitative and quantitative methods.
The fun part came when we asked NotebookLM to create slide shows, quizzes, and infographics. They were created in seconds. We used the slide shows to facilitate evaluation training sessions to build their evaluation capacity. The repository became another resource for them to ask their own questions and make their own podcasts, slide shows, reports, mindmaps, and data sorts. We plan on continually building and updating the repository by adding additional sources.
Rural Senses. The next step included introducing The Just Trust to Rural Senses, a fully integrated AI-driven evaluation platform. Communities and agencies upload their data to the Rural Senses platform. It produces a draft logic model (which the group can edit and correct), initial and impact findings, and draft reports (in Google Doc format).
We invite our colleagues to apply some of these AI tools to enhance their own AI evaluation capacity and the capacity of the communities and organizations they work with daily.
Rad Resources
- EmpowerEval Assistant (empowerment evaluation GPT)
- https://chatgpt.com/g/g-Db8NmkXGd-empowereval-assistant
- GPT Creator
- https://chatgpt.com/gpts
- Google’s NotebookLM
- https://notebooklm.google/
- ImpactMate
- https://prove.impactmate.ai/
- Rural Senses
- https://ruralsenses.com/
- Gestalt Empowerment Evaluation: A Culturally Responsive AI-Driven Evaluation Approach, in the Handbook of Culturally Responsive Evaluation and Culturally Responsive Research (forthcoming)
Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Hi Jonathan
Words are important; I agree.
Thanks for the opportunity to expand on my wording and my thoughts concerning Gestalt Empowerment Evaluation. Gestalt in this case has to do with the way we conduct empowerment evaluation. Instead of a linear 3-step or 10-step approach, we “go with the flow” of an organization’s decision-making process. It is much more organic and feedback is more relevant because it is in sync with their organizational decision making ebbs and flows. Dr. Paul St. Roseman and I will be sharing more about that in our AEA presentations and workshops and in a forthcoming book. Our chapter will be Gestalt Empowerment Evaluation: A Culturally Responsive, AI-enhanced Evaluation Approach. You note we use the word enhanced here. It will appear in Frierson, Hall, Boyce, and Hopson (eds.) Handbook on Culturally Responsive Evaluation and Culturally Responsive Research. Oxford Press (in press).
Our work is sometimes AI-informed, as you suggest. See our posting AI-Informed Empowerment Evaluation by David Fetterman and Paul St. Roseman. It is at: https://aea365.org/blog/ai-informed-empowerment-evaluation-by-david-fetterman-and-paul-st-roseman/
However, there are times when it is AI-Driven and that is why we highlight the tools in the posting because it sharpens our evaluative thinking and allows communities to see patterns in complexity (once the findings are validated). It helps us see wholes rather than isolated elements. In many communities and organizational settings, evaluative data do not arrive as clean linear signals. Information may be uneven, contradictory, missing, or shaped by historical inequities and power relations. Fragmentation can make it difficult for program staff and community members to see how activities, outcomes, and contextual forces relate to one another. This is when our work can be AI-Driven. I am happy to expand on my thoughts about this subtle distinction in wording offline or you can see what we are talking about in more depth at AEA this year or in our forthcoming chapter. Many thanks for the opportunity to expand our use of language. We agree words matter. Best wishes Jonathan.
Hi David and Paul
Because words are important (e.g. Gestalt), I have to push back on something in this blog. It is a mistake to frame evaluation work (or its implementation) as AI-driven. Like email, Excel, and online survey collection, AI is a tool. AI (and specifically LLMs) should not drive any part of our work.
As someone who is extremely AI-positive, I believe the use cases of LLMs only are valuable in so much as there is a sensemaking layer between input –> model and output –> intepretation. “AI-informed” is a much more useful and accurate phrase.