Hello! Welcome to the Needs Assessment (NA) TIG Week on AEA365. As the AEA proposal process for the 2024 AEA conference in Portland is underway, the NA TIG invites you to submit a proposal, volunteer to be a proposal reviewer, or simply join our TIG in your AEA profile. Reach out to our TIG Chair Lisle Hites for more info.
I’m Ryan Watkins and I’m a professor at George Washington University, program director of the Educational Technology Leadership (MA) program, and faculty lead of the interdisciplinary Human-Technology Collaboration (PhD) concentration (research lab). I’ve authored 12 books and over 100 articles/chapters, and developed several web platforms (including Code2Learn, We Share Science, SciencePods). I also maintain www.needsassessment.org.
Over the last year it’s been hard to ignore the attention given to Artificial Intelligence (AI), especially with Generative AI (i.e., AI that can create) advancements such as Large Language Models (LLMs) like OpenAI’s ChatGPT and DALL-E, Google’s BARD, Stability AI’s Stable Diffusion, and more recently Phind. These tools have garnered significant attention for their growing capabilities to perform tasks such as creating images, writing songs or college essays, editing draft reports, and generating computer code. As an evaluator, researcher, and daily user of LLMs I’ve been interested in where and how these tools will get applied. Here are some practical suggestions on how to use Artificial Intelligence (AI) tools in Monitoring & Evaluation (M&E), and more specifically Needs Assessments.
Currently, we interact with LLMs through “prompts”, which are simply statements of what we want LLMs to do for us. These can be questions to be answered, requests for images to be created, or complex computer code that you want written. Writing a quality prompt is therefore different from writing a quality Google search inquiry. Here are some tips for writing better prompts:
Lessons Learned
- Tell the LLM the character that you want the system to embody when responding. For example, “You are the CEO of an international nonprofit organization…”
- Include details about what you want in the response. For example, “Respond with at least 6 positive outcomes, 6 challenges, 10 recommendations for future work.”
- Give multiple prompts. LLMs aren’t people and won’t be offended if you request many of whatever you want. For example, “Provide me with 40 possible questions to be included in my survey on….”
- Break down a complex request into smaller prompts to create an interactive conversation. For example, “To get started, your first task is to identify 5 indicators that can be used to assess…”
Rad Resources
- How to Learn and Teach Economics with Large Language Models
- Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4
Hot Tips
After experimenting with a variety of prompts, next you will have to decide how to use LLMs in support of your specific M&E or Needs Assessment work. Here’s a list of some uses I have found helpful:
- Ideation
- Use multiple LLMs as brainstorming partners.
- Ask the LLM to provide counter-arguments to your ideas.
- Background
- Upload and ask questions about a document. (try: ChatPDF)
- Have the LLM translate articles in other languages.
- Writing
- Synthesize a few paragraphs at a time.
- Edit for grammar, vocabulary, clarity
- Data Preparation
- Convert text or numbers into various formats.
- Extract specific data from textual content.
- Data Analysis
- Classify and score text.
- Create synthetic/dummy data.
- Presentation
- Create charts or figures.
- Generate unique images based on content.
- Coding
- Write computer code.
- Explain computer code.
Rad Resources
AI and LLMs can be game-changers for your needs assessment or M&E work. It’s like having a supercharged assistant by your side, ready to brainstorm, analyze data, or even help draft reports. Start today with some fun tasks (e.g., write a poem about M&E) and try to integrate into your work routines to gain small efficiencies (e.g., draft an email to participants). Get creative, keep experimenting, and have fun.
The American Evaluation Association is hosting Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to AEA365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.