Hello, AEA365 community! Liz DiLuzio here, Lead Curator of the blog. This week is Individuals Week, which means we take a break from our themed weeks and spotlight the Hot Tips, Cool Tricks, Rad Resources and Lessons Learned from any evaluator interested in sharing. Would you like to contribute to future individuals weeks? Email me at AEA365@eval.org with an idea or a draft and we will make it happen.
Hi! We are Andrew Grillo-Hill, Joshua Valcarcel, Kimberly Nguyen, Karen Melchior, Corynn Del Core, and Rasha Elsayed, the N3 evaluation team within WestEd’s STEM Evaluation Unit. The STEM Evaluation Unit serves as external evaluators for dozens of STEM education projects and programs each year, including for multiple NASA Science Mission Directorate Science Activation 2.0 projects, or SciAct for short.
As evaluators, we wear the hats of being critical friends and facilitators. Today, we would like to share with you one tool we use to share leadership and decision-making responsibilities with our program partners: the Evaluation Debrief tool and process.
We developed this process while serving as external evaluators for NASA’s Neurodiversity Network (N3), a five-year program that provides pathways to NASA participation and STEM employment for neurodivergent learners. N3 involves multiple partner organizations, so transparent communication and shared decision-making were essential from the start.
Here is how the Evaluation Debrief process works. First, we collect and analyze our evaluation data and draft the report. Then, we fill out a debrief form that summarizes key findings alongside our recommendations. We share both the report and the form with our program partners, giving them time to process the findings. Next, we schedule a debrief meeting with relevant partners to discuss and document their reactions and collaboratively decide on the next steps directly in the form. Each of these collaborative and formative meetings has ended with at least one call to action which provides a new iteration of the program for the next cohort, and the cycle begins again.
To illustrate, in Year 1 of the project, our evaluation found that mentors needed more student-specific guidance given the wide range of needs among neurodiverse interns. Through the debrief process, program partners decided to implement “intake meetings” with each intern, their mentor, and an N3 staff member in addition to bimonthly check-ins. This concrete change came directly from the structured dialogue facilitated by the debrief, not from evaluators telling partners what to do, but from partners making informed decisions based on evaluation data.
Rad Resource:
We are sharing a template of our Evaluation Debrief Feedback Form that you can adapt for your own projects. The form includes three columns: Evaluation Recommendation/Finding, PI Response/Question, and Action Taken, along with reflection questions about the timing and usefulness of the findings to give us –the evaluators –feedback too.The form, by design, is simple and concise to allow for less more open-ended and less “dictated” conversations during the debrief discussions.
A PDF of the first page of the form, with two paragraphs of instructions for program partners.
A PDF of the second and last page of the form with a blank table to fill out and two open-ended questions for feedback on the evaluation.
Lessons Learned:
As the N3 program nears our end date this year, we have been able to look back at how this process shaped the partnership that allowed for a program that was so rewarding to be a part of. Three things, especially, stand out.
- The form created a documented trail of data-informed decision-making that partners could revisit over time.
- Transparent documentation of the decision-making built trust across organizations.
- The process gave program partners collective ownership over program modifications.
Overall, the Evaluation Debrief process contributed to more efficient, equitable, and transparent shared leadership throughout the life of the project.
Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.