GAO Week: GAO’s 100th Celebration: Examples of How Agencies have Addressed Some Evaluation Challenges by Elaine Vaurio

Hello, I am Elaine Vaurio, a Senior Analyst with the Center for Evaluation Methods and Issues (CEMI) in the Applied Research and Methods (ARM) team at the U.S Government Accountability Office (GAO). I have proudly been associated with evaluation at GAO for 30 years!  GAO has produced many hidden gems over the years that I want to highlight for you!

Agencies and non-profits face various challenges to assess program performance and evaluate outcomes. Some challenges they face include hard to measure outcomes due to program design, the need to explain program performance to stakeholders and funders, and the challenge where a sound methodology may still not produce clear answers. I note some GAO reports that address such challenges with solutions that you may find relevant and useful.

Hot Tips:

There are ways to evaluate programs that by design, use information to persuade others to change their individual behavior to achieve desired agency outcomes. For example, anti-drug media campaigns, food nutrition education, and tobacco control programs seek to provide information to improve physical well-being. .

• Tip – Surveys of post-program awareness can be logically connected to agency outcomes with logic models, but external factors still need to be addressed, see [Info Dissemination].

Sometimes, we have information on program performance, but need to explain it to stakeholders and funders (legislative or private).

• Tip – A complex evaluation can conduct multiple analyses to answer distinct questions and, thus, serve several purposes in a performance report. [Worker Assistance]
  • Tip – Evaluation studies can help develop and pilot test measures of client status and outcomes. [Explain Program Performance].
  • Tip – For oversight and funding decisions, legislators need descriptive information about the program, and information about intended and unintended effects, so they can understand the variety of conditions under which a program operates, the kind of coverage a targeted program has achieved, and how federal funds are spent. [Information to Congress].

A sound methodology can help identify effective programs and assess impact, but there is no guarantee clear answers will be obtained.  

• Tip – An Evaluation study design can help examine effectiveness even where there are complex federal initiatives and where desired outcomes are infrequently observed [Designing Evaluations].

• Tip – Even with well-conducted randomized experiments for assessing effectiveness when multiple causal influences create uncertainty about what caused results—study controls may not work, external factors may not have been assessed, or the results were not as expected. [Worker Assistance].
  • Tip – Other rigorous alternative methods to randomized experiments are considered appropriate in certain situations (e.g., quasi-experimental comparison group studies, statistical analyses of observational data, and in-depth case studies). The credibility of the estimates of program effects relies on how well the study design rules out competing causal explanations.
    • Collecting additional data and targeting comparisons can help rule out other explanations. [A Variety of Rigorous Methods Can Help Identify Effectiveness Interventions]. 
    • Some alternative methods have been applied to [abstinence education], [violence against women], [teen-mom education], and [early childhood education].  

I hope I have identified some useful resources that are free, produced by experts, free of political bias, and have met the test of time. These are gems in education, justice, and human services.

Lesson Learned:

There is no one way to think about evaluation. GAO has focused on evaluation from the federal government perspective, but we can learn from one another and apply tools, methods, and concepts to the range of current challenges.

The American Evaluation Association is hosting GAO Week in celebration of the US Government Accountability Office’s 100^th anniversary. The contributions all this week to aea365 come from authors who address GAO’s efforts to solve those complex “wicked” socio-cultural problems that defy permanent solutions but demand our best efforts to solve them. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.