Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hi. I am David Fetterman. I am the president of Fetterman & Associates, an international evaluation firm. I am also a past-president of the American Evaluation Association and a co-chair of the Collaborative, Participatory, and Empowerment Evaluation TIG. I am the founder of empowerment evaluation and I see some parallels between empowerment evaluation and the artificial intelligence tool ChapGPT.
Empowerment Evaluation
There was a lot of controversy about empowerment evaluation when it was introduced during my president address. Some thought it was the best thing since sliced bread, others feared it was giving evaluation away. It is now a part of the evaluation landscape.
The evaluation approach is described in:
- Books:
- Empowerment Evaluation and Social Justice: Confronting the Culture of Silence
- Empowerment Evaluation: Knowledge and Tools for Self-assessment, Evaluation Capacity Building and Accountability
- Collaborative, Participatory, and Empowerment Evaluation: Stakeholder Involvement Approaches
- Empowerment Evaluation: Principles in Practice
- Empowerment Evaluation in the Digital Villages: Hewlett-Packard’s $15 Million Race Toward Social Justice
- Foundations of Empowerment Evaluation
- Articles (online):
- Blogs: Better Evaluation, Empowerment Evaluation
- Videos: Empowerment Evaluation Ignite Lectures
- Wikipedia: Empowerment Evaluation
Just as there was a lot of controversy about empowerment evaluation and it has slowly been sewn into the social fabric of the field, there is a lot of controversy about ChatGPT and it too may become a part of how we help stakeholders build evaluation capacity.
ChatGPT
ChatGPT is an online artificial intelligence program designed to generate human-like text responses based on prompts submitted by a user. For example, the following prompt “what is empowerment evaluation” generated the following response:
“Empowerment evaluation is a type of evaluation approach that prioritizes the active participation and empowerment of stakeholders in the evaluation process. It differs from traditional evaluation methods in that it places a greater emphasis on building the capacity of stakeholders to drive their own development and assess their own progress.
In empowerment evaluation, stakeholders are involved in setting evaluation goals, collecting and analyzing data, and using the results to make decisions and take action. This approach recognizes that stakeholders are the experts on their own situation and are best positioned to identify what works and what doesn’t in their own context.
Empowerment evaluation also emphasizes capacity building, so that stakeholders have the skills and resources they need to continue to drive their own development after a program has ended. The ultimate goal of empowerment evaluation is to promote sustainable, community-led change.
This approach is often used in community development programs, education initiatives, and health interventions, among other areas.”
The strength and potential of ChatGPT is that it can process large amounts of data and respond quickly and accurately. It can help people get a jump start on learning about a topic, help them do a deep dive into a topic, and make links to a variety of interrelated topics, including empowerment evaluation and systems thinking. It can save time and increase efficiency in these areas.
The concern is that it will be used to help people “cheat” (instead of doing the work themselves); reduce opportunities to problem-solve; and like the introduction of calculators reduce the need for people to compose their own thoughts.
However, if used wisely and with a sound education policy and practice, ChatGPT can be used to help people learn about evaluation, conduct an evaluation of their own efforts, and plan for the future. They can use ChatGPT to ask formative and summative evaluation questions. A few evaluation questions for ChatGPT are presented below:
- “Create a rubric that stakeholders can use to self-assess their work on [specify task task].”
- “Create a set of questions or tasks that community members can use to reflect on and apply the feedback they received on their [specify task].”
Rad Resources
Lessons Learned
We probably should not dismiss or condemn a new approach to learning or evaluation just because it is new and seems threatening. It may be wiser to bring it into the fold and help guide its application to the mainstream to maximize its effectiveness.
David Fetterman is the recipient of the following AEA awards: Myrdal Award for Practice, the Lazarsfeld Award for Theory, and the Evaluation Advocacy and Use Award.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
The pace of AI is impossible to keep up with. The new Chat4 GPT can take one of your pictures of the food in your refrigerator and generate recipes for you, write SAS code, and transform a sketch on a napkin into a web page. The risk of fraud is also there because it can be used to hijack Facebook pages. However, I still focus on how it can be used “for good” and am pulling a session together with John Baek (chair of the Integrating Technology in Evaluation TIG) for the next AEA meetings. Susan Wolfe will serve as a discussant. Feel free to post or email your comments and examples of how you have used it in evaluation in preparation for the session.
Hi Jeremy
Yes – there are many problems associated with ChatGPT in its current configuration. I would recommend Wordtunes if you are looking for something with reliable citations and verifiable accuracy of statical facts.
Both, however, have real advantages for minimizing much of our busy work. My colleague Susan Wolfe has used ChatGPT to write SAS code. It is also a great way of organizing initial thoughts about what’s out there – much like a literature review helps us know what’s out there before jumping into a project.
Thanks for the critical comments – very open to them as we refine our understanding of something that is potentially very useful and certainly here to stay. Best wishes. – David
Thanks, David. I hope my comments are understood as “critical” in that I’ve read the post and reflected on the issues. In raising these points, I’m trusting the experts that know/have the technical background and are asking relational and evaluative questions.
In making an alternate suggestion, I think you’re misunderstanding my point. There’s no evidence that any of these platforms address the foundational concerns I referenced. With regards to accuracy, they will confidentially produce garbage data and non-existent references. If I have to verify each one, what’s the point of using them?
What is their corpus? How was it generated? How was it trained? On what data? Was that data stolen, as many of the creators themselves admit? There’s no way to know the extent of these questions. To me, this is the very opposite of personal empowerment, though I’m sure it’s (further) empowering some people.
Thinking of the environmental effects it takes to run powerful computers to generate all that text/content, it’s crypto all over again on that front.
To your point of SAS code, I’ve also seen them generate completely unusable code in a wide variety of programming languages, but I’m sure they spit out good code sometimes! It’s bound to happen, just as these mistakes. Yet, I’d just rather write it myself or go to trusted, transparent sources than have to check everything.
For what it’s worth, I don’t accept that any of them are here to stay, unless you’re referring to a very broad understanding of both “stay” and these salami “AIs.” As it stands, it’s difficult for me to envision anything but harm based on the systems and history at work. Then again, that has been historically the type of thing that stays…so maybe you’re right. But it has no place in my practice.
Thanks again Jeremy. I appreciate the dialogue and discussion on the topic. There is a conference on the topic of AI and Evaluation coming up on March 24th that might interest you. In any case, thanks again for your candid comments concerning these AI tools. Best wishes. – David
ChatGPT and it’s like are not AI. At best, they are “salami.”
https://blog.quintarelli.it/2019/11/lets-forget-the-term-ai-lets-call-them-systematic-approaches-to-learning-algorithms-and-machine-inferences-salami/
In my view, it’s the worst salami – mass-produced, prepackaged, unsourced, extractive, and mixed with no attribution from stolen material – served up with little fuss in a neat package (as long as we don’t critically examine it).
As others smarter and more involved than me have noted (and refuting the claims in the blog post above), it’s also been programmed to be unreliable and inaccurate. It’s programmed to make up citations.
Honestly, it’s a bit baffling to me to read/hear that so many evaluators are promoting it, and that they don’t seem aware of the issues discussed at length by experts for years (as well as in this current moment). Even so, I’m hoping that these discussions will help lead to greater awareness of the many, many severe issues.