Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
My name is Dr. Kinsey Simone, and I teach quantitative research methods at Tennessee Tech University and evaluate university-internal and external projects. I never look forward to opening a blank page to draft an evaluation report based on (usually) a large plethora of data points. In this blog, I share three rad (and free!) resources regarding Artificial Intelligence (AI) platforms that can make this task more efficient without losing accuracy.
In April 2024, I planned and implemented a Mad Topics panel symposium on obsessive-compulsive disorder (OCD) and attention-deficit hyperactivity disorder (ADHD) in education, which provided a free opportunity to hear clinical perspectives of mad allies and lived experiences of mad-identified panelists. An exit survey assessed overall effectiveness of this event. While I did not use AI to draft my original report, I was curious as to whether AI would come to the same conclusions as I did regarding the evaluation of this event. I compare these rad AI resources and how they related to my own findings below.
What I (the human) found:
Exit survey results indicated that the panel was overall effective in increasing knowledge, awareness, and empathy in attendees for OCD/ADHD. The symposium attracted a diverse group of students, educators/faculty, community members, parents, and mental health professionals, with a mix of diagnosed and non-diagnosed persons. Suggestions for improvement included a longer panel and more one-on-one time with panelists in the post-panel Q&A.
Rad Resource 1: ChatGPT
ChatGPT is an advanced AI platform developed by OpenAI. I uploaded a Word document including the raw frequencies of all assessed variables and a table of qualitative responses from the exit survey and prompted ChatGPT to: “Summarize the key findings from this document for evaluation.” A report was generated quickly and included an accurate overview of the document; statistical evidence and a summary of increased knowledge, understanding of lived experience, empathy and compassion, event satisfaction, and suggestions for improvement; and a summary stating that the event was successful. The information paralleled with my own report, and stats were accurate.
Rad Resource 2: PopAi
After using the same prompt in PopAI, an advanced AI platform, a report like ChatGPT was rapidly generated and included an introductory and concluding statement and six key points. Unlike ChatGPT’s emphasis on attendees’ perceptions within various categories, PopAi provided a more holistic report which included educational applications, panel feedback, event structure and satisfaction, key takeaways, improvement suggestions, and informed consent. While PopAi’s report highlighted qualitative responses, it was general and did not provide any statistical evidence.
Rad Resource 3: AI Word Doc Summarizer (screenapp)
Rather than using prompts, the AI Word Doc Summarizer on screenapp only requires a file upload before generating a summary, making it a great tool for getting narrative descriptions of each frequency within a categorical variable. It would not prove helpful in generating a readable, human-like report. This tool generated a report that simply provided each variable/topic name, such as “Compared to BEFORE the event, overall understanding of ADHD is…” along with the percentages and ratios of responses to each of the Likert categories.
Conclusions
In this blog, I shared about three rad resources which can generate quick and accurate evaluation reports from raw frequency tables. ChatGPT and Pop AI can assist in the creation of human-like reports, while screenapp is better suited for statistical narrative description without interpretation. While the world of AI might be intimidating at the beginning (it is for me!), using free platforms to summarize data for evaluation is a great way to get started.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.