Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hi. I am David Fetterman. I am the president of Fetterman & Associates, an international evaluation firm. I am also the past-president of the American Evaluation Association and a co-chair of the Collaborative, Participatory, and Empowerment Evaluation TIG (with Liliana Rodríguez). I am the recipient of the Myrdal Award for Practice, the Lazarsfeld Award for Theory, and the Evaluation Advocacy and Use Award. I am also the founder of empowerment evaluation.
AI
AI is based on large language models. They calculate the probabilities of the next letter and word. For example, it calculates that the next word after “United States of” will be “America.” It applies the same logic to paragraphs, documents, and images. AI is driven by prompts (commands and questions). Like to a Google search, the more precise the prompt the more precise the product or result.
AI Evaluation Examples
AI can guide and enhance evaluation practice, including helping build evaluation capacity in our schools, communities, and organizations. Examples based on these prompts include:
- Provide a logic model for an evaluation of a tobacco prevention program
- ChatGPT accurately listed inputs to impacts
- Provide a list of interview questions in an empowerment evaluation
- Bard provided a list of useful questions and added tips ranging from create a safe and supportive atmosphere to be respectful of different perspectives
- Create a picture of a globe with a cigarette on it (helping highlight the impact of tobacco use worldwide in a tobacco prevention evaluation)
- Image Creator (driven by Dall-e software) created an image of a globe with a cigarette sticking out of the globe
- Create an image of the AI moat (the width of the gap between AI industry and individuals concerning access to AI)
- Image Creator (powered by Dall-e software) produced a picture of big AI technology companies on an island with a moat around the island and individuals standing around the moat. This picture was used to highlight how much smaller the moat is getting at a recent AEA annual meeting presentation about AI and evaluation (people filled the room, sat on the floor, and stood in the hallways)
AI Evolution: From the Cloud to Desktop to Smartphone
The next evolution of AI involves helping organize, synthesize, and summarize files on our desktop computers and smartphones. It can sort and synthesize content in our emails and private drives (without sharing it on the net with AI learning systems). For example:
- Dropbox’s Dash can search for all the files concerning birthday or empowerment evaluation
- Google’s Bard with Google Assistant can summarize all the documents concerning empowerment in your files or find a flight that gets you back home based on your email correspondence
Caveats and Responsibilities
I recognize many of the problems associated with using AI. “Hallucinations” or inaccurate information, for example, require triangulation or accuracy checks. AI programs are increasingly including a “button” to facilitate accuracy checks and provide valid bibliographic sources. Racial and gender bias, based on the input or sources of data, require the same due diligence we exercise in our evaluation work. Additional prompts can be used to help identify these problems and, in some cases, correct for it.
These are challenges, not barriers, for evaluation. I think we have an obligation to enter the AI space to identify and name the problems as we see them. I also think we should mine AI for what it has to offer the field. It is an opportunity and our responsibility to help guide its application to minimize misuse and maximize its effectiveness.
Rad Resources
Dall-e and Image Creator
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.