GovEval TIG Week: Building Evidence with Rigorous Evaluation – Lessons from Federal Partnerships by Michael Hand

Hello, I’m Michael Hand, a Portfolio Lead and Behavioral Scientist with the Office of Evaluation Sciences (OES) at the U.S. General Services Administration (GSA). At OES, our staff of interdisciplinary evaluation experts works within GSA and collaborates with agency partners across the government – including the Small Business Administration, Environmental Protection Agency, the Department of the Treasury, and many others – to build and use evidence to tackle the most pressing problems in the United States. Conducting rigorous evaluations from within the federal government is a unique challenge and exciting opportunity to improve the lives of Americans. We’ve learned that building evidence with our agency partners starts with a commitment to transparency, rigor, and close collaboration.

Lessons Learned:

Collaborating with partners helps us design the best possible evaluations. None of us can be experts in every policy and programmatic topic across the federal government. To deliver actionable evidence, we rely on the subject matter expertise of our partners to tailor evaluations to their needs and priorities. For example, OES has collaborated with the Internal Revenue Service (IRS) to reduce improperly claimed refundable tax credits (focusing on tax preparers and their clients), resulting in millions of dollars saved for U.S. taxpayers. These evaluations were possible because of a close collaboration with IRS staff who understood the agency’s complex outreach and filing processes. OES brought our evaluation and methodological expertise to design evidence-based interventions and evaluations that could complement existing efforts to reach tax preparers and filers.

Designing evaluations to inform priority decisions can lead to impact. When the 988 Suicide and Crisis Lifeline made changes to their automated phone system in 2023, rigorous evaluation was a priority to support continuous improvement. We conducted a cluster-randomized evaluation to learn whether the changes resulted in more callers being connected to counselors. The phone system changes did work, with about 1,400 additional callers connected to counselors during the four-week study, and the evaluation helped inform additional changes to improve the experience for 988 callers.

A commitment to transparency drives rigor, reliability, and trustworthiness. Informing decisions that our agency partners need to make is a primary objective for our work. Our partners need to be confident that the evaluation results are reliable and trustworthy. Two steps in our project process help us do this: First, we post analysis plans on our website before we access data, ensuring that how we conduct an analysis isn’t influenced by the potential outcomes. Second, we publish the results of all our evaluations, regardless of the findings, so others can learn from all of our work. For example, publishing the analysis plan and results of a randomized evaluation to increase Supplemental Security Income uptake among eligible older adults allowed us to collaborate on a follow-up evaluation showing that enrollment decreased mortality.

Rad Resources:

On our website we share the OES evaluation process that helps us deliver value and build evidence for our partners. Templates of our project process documents, such as the analysis plans, can be used by others looking to incorporate rigorous evaluation in the federal government.

---

The American Evaluation Association is hosting Gov’t Eval TIG Week with our colleagues in the Government Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our Gov’t Eval TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.