My name is Valerie Jean Caracelli, and I am a Senior Social Science Analyst in the Center for Evaluation Methods and Issues, Applied Research and Methods team at the U.S. Government Accountability Office.
As we greet the 5th year anniversary of the Foundations for Evidence-Based Policymaking Act of 2018, it is useful to reflect on federal evaluation and its use in decision making prior to the passage of the Evidence Act. In 2013 a series of evaluation questions were introduced into a generalizable survey of federal civilian managers and supervisors to obtain their perspectives on several results-oriented management topics, including the extent of and barriers to evaluation use. The survey results indicated just over a third (37 percent) of federal managers reported that an evaluation had been completed in the past 5 years on any program, operation, or project they were involved in. GAO concluded that agencies’ lack of evaluations may be the greatest barrier to their ability to inform program management and policy making.
When GAO repeated the survey in 2017, familiarity with evaluations was still low. Just 40 percent of managers reported being aware of evaluations conducted over the previous five years, while 39 percent did not know whether an evaluation had been conducted. GAO did learn that those who conducted evaluations found them useful: half the managers who had evaluations reported that those evaluations contributed greatly to assessing program effectiveness or value and improving program management or performance. Respondents frequently reported that agency leadership support for evaluation, staff involvement, and an evaluation’s relevance to decision makers helped to facilitate evaluation use. However, lack of resources to implement the findings was the most frequently reported barrier to the use of evaluations.
In the 2017 report, GAO recommended that the Director of the Office of Management and Budget (OMB) direct the 24 CFO Act agencies to develop an agency-wide annual evaluation plan in consultation with congressional and other program stakeholders to help ensure agencies obtain the evidence needed to address the most important questions to improve program implementation and performance. That recommendation was implemented with the passage of the Foundations for Evidence-Based Policymaking Act of 2018 and the issuance of OMB memorandum M-19-23.
In 2020 after the passage of the Evidence Act (in 2019), GAO modified the Federal Managers Survey to provide a focused look at the factors that predicted evaluation use. With a 62% survey completion rate, we found respondents cited elements of the following factors as predictors of use: the quality of evaluations, human capacity for evaluations, and other facilitating factors for use, including leadership support, which was consistent with results from GAO’s earlier surveys and work. The current version of the Federal Managers Survey provides baseline information about perceived use of evaluation by federal managers at an early stage of Evidence Act implementation. In a future federal manager survey, we can address whether the snapshot changes as the implementation of the Evidence Act becomes mature. The results of the most recent Federal Managers Survey hold early promise as we explore trends on the use of evaluation and evidence throughout the week.
Rad Resources
- Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making, GAO-13-570, Jun. 26, 2013.
- Program Evaluation: Annual Agency-Wide Plans Could Enhance Leadership Support for Program Evaluations, GAO-17-743, Sept. 29, 2017.
- Evidence-Based Policymaking: Practices to Help Manage and Assess the Results of Federal Efforts, GAO-23-105460, July 12, 2023.
The American Evaluation Association is hosting Washington Evaluators (WE) Affiliate Week. The contributions all this week to AEA365 come from WE Affiliate members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.