I’m John Baek, Senior Education Evaluator at the National Oceanic and Atmospheric Administration and chair of the Integrating Technology in Evaluation (Tech) TIG. Along with Noah Goodman (Wednesday’s author), I’m curating this week’s AEA365 blogs from evaluators who are actively exploring the use of artificial intelligence tools as part of their evaluation practice.
Tools like ChatGPT have sparked the imagination of many evaluators. This week’s posts will share some of our colleagues’s explorations of these tools with their lessons learned and hot tips that cover the following topics:
- Experimenting with everyday uses of genAI tools in evaluation practice
- Presenting case studies of using a genAI and machine learning tools for analysis
- Exploring the human dimensions of AI tools
Lessons Learned
As TIG chair, I’ve witnessed the growth of AI and Evaluation since 2022, when ChatGPT and DALL-E first burst on the scene. If you thought we hit the peak of the hype cycle last year, I think we’re still on the upswing as a professional community.
I wanted to share some data that provides one facet of where we are in the state of AI in our field. The Tech TIG received 44 proposals for the conference to review this year, of which 27 (over 60%) included the topics of artificial intelligence, large language models, natural language processing, machine learning, and ChatGPT. Compare that to last year where we received 5 proposals out of 23 (21%) with those topics.
With all of the buzz, not everyone is using these tools everyday, or at all. I admit, I don’t use these tools everyday (partly because my agency regulates/restricts its use). I also know that there are evaluators out there who feel strongly against AI’s use in evaluation. It’s important to remember that all technologies have the potential to be intentionally and unintentionally harmful. Vigilance and skepticism help counterbalance our unbridled enthusiasm.
Lastly, even those of you who might never open up one of these AI chatbots will be confronted with AI that is integrated into your everyday tools, like Word and Gmail. Increasingly data analysis software, both qualitative and quantitative, now include AI-enhanced capabilities. The AEA Guiding Principle of Competence means we should ensure that we have the knowledge and experiences required to make judgements about how and when we integrate AI in our work.
Hot Tip
We saw standing room only crowds at genAI sessions at the AEA conference last year. And we’ll likely see them again this year, so get to the room early.
Get Involved
To learn more about what other evaluators are learning and doing with regard to emerging technologies, visit the Tech TIG’s website. Join the TIG to hear about AI-related events at the conference this year.
If you want to go deep on generative AI tools, check out MERLTech’s NLP Community of Practice as well as Zach’s blog post tomorrow.
Finally, you can join the AI and Evaluation LinkedIn group to connect with others interested in AI.
The American Evaluation Association is hosting Integrating Technology into Evaluation TIG Week with our colleagues in the Integrating Technology into Evaluation Topical Interest Group. The contributions all this week to AEA365 come from ITE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.