Welcome to aea365! Please take a moment to review our new community guidelines. Learn More.

DVR TIG Week: Want to Control the Quality of the Data You Inquire About? Make Your Own AI Chatbot by David Fetterman

Hi.  I am David Fetterman.  I am the president of Fetterman & Associates, an international evaluation firm. I am also the past-president of the American Evaluation Association and a co-chair of the Collaborative, Participatory, and Empowerment Evaluation TIG (with Liliana Rodríguez). I am the recipient of the Myrdal Award for Practice, the Lazarsfeld Award for Theory, and the Evaluation Advocacy and Use Award. I am also the founder of empowerment evaluation (and a proud dad whose son is going to Northwestern this year and daughter who is an established, award-winning artist).

AI and Chatbots

Artificial intelligence is certainly the rage. It is something as evaluators we need to understand and engage.  In any case, it is here to stay (whether we like it or not).

There are many AI chatbots to help us with theories of change, logic models, interview questions and data analysis, including ChatGPT, Claude, and Microsoft Copilot AI. AI image creation tools are also helpful including Dalle-3, KREA, and many others. They can produce powerful visual images to increase the impact of our findings and recommendations.

They can all be powerful tools to assist us as evaluators.  However, it has already been established they can have built-in biases, misalignments with social values, and a periodic propensity to hallucinate. 

What are We to Do?  One solution is to Make your Own Chatbot!

I test chatbots routinely. The results are generally impressive.  However, in specialized evaluation areas, they can produce generic results.  In addition, the information is often incomplete.  I used a variety of prompts to inquire about empowerment evaluation to test various chatbots.  It produced excellent responses concerning the founder, the basic definition, interview questions, and so on.  However, it lacked the depth I was looking for as a practitioner.  In a few cases, it produced misleading information.

I decided to do something about it. I learned to create my own GPT.  The rationale for building one was two-fold:  to ensure quality data and to learn how they work placing me in a better position to evaluate them.

Steps to Creating Your Own GPT

Creating your own GPT is not hard.  You can use GPT Creator or GPT Builder. It will walk you through the required steps.  It will help you generate instructions to use your GPT, provide suggested names, create a profile picture, and most importantly – attach the files you think are appropriate to build your database (see Figures 1-8 below).

Figure 1. EmpowerEval Assist GPT
Figure 2.  Creating Your Own GPT
Figure 3. Generating Instructions
Figure 4. Naming Your GPT
Figure 5. Creating a Profile Picture
Figure 6. Posting Your GPT
7. Uploading Files to Your GPT
8. Asking Your GPT Questions After Uploading New Files

Building your own database is a useful exercise.  First, it provides people with quality, vetted data.  The results of their inquiry are more likely to be valid and useful.  Second, it minimizes the inclusion of poor-quality or erroneous data.

Maintaining your own GPT is an iterative and evolving process. After creating my own GPT, I tested it. I asked for videos of empowerment evaluation.  It said they existed but did not point me to them.  I could have produced better results with a Google search.  I rectified the problem by uploading a Word document.  It provided my GPT with a variety of videos about empowerment evaluation, including a recent TED talk.  After including relevant data, I ask the same question.  This time it produced the desired results – a list and link to relevant videos.

Try It

My only recommendation is to play with the AI tools to learn more about how to use them in evaluation.  You will learn to improve your prompts just like we learn to improve them with Google searches.  Learn more about chatbots so you too can learn how they work and be in a better position to evaluate how they work.  It will improve your daily practice and possibly safeguard us concerning AI misalignments and social values in our efforts to link evaluation to social justice issues.

Rad Resources

  • AI Tools: GPT Creator, GPT Builder, ChatGPT, Claude, Gemini, and Microsoft Copilot A
  • TED Talk: Empowerment Evaluation:  Searching for Self-determination and Results
  • Book: Empowerment Evaluation and Social Justice: Confronting the Culture of Silence

The American Evaluation Association is hosting Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to AEA365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.