GAO Week: GAO’s 100th Celebration: Examples of How Agencies have Addressed Some Evaluation Challenges by Elaine Vaurio

Elaine Vaurio
Elaine Vaurio

Hello, I am Elaine Vaurio, a Senior Analyst with the Center for Evaluation Methods and Issues (CEMI) in the Applied Research and Methods (ARM) team at the U.S Government Accountability Office (GAO). I have proudly been associated with evaluation at GAO for 30 years!  GAO has produced many hidden gems over the years that I want to highlight for you!

Agencies and non-profits face various challenges to assess program performance and evaluate outcomes. Some challenges they face include hard to measure outcomes due to program design, the need to explain program performance to stakeholders and funders, and the challenge where a sound methodology may still not produce clear answers. I note some GAO reports that address such challenges with solutions that you may find relevant and useful.

Hot Tips:

There are ways to evaluate programs that by design, use information to persuade others to change their individual behavior to achieve desired agency outcomes. For example, anti-drug media campaigns, food nutrition education, and tobacco control programs seek to provide information to improve physical well-being. .

  • Tip – Surveys of post-program awareness can be logically connected to agency outcomes with logic models, but external factors still need to be addressed, see [Info Dissemination].

Sometimes, we have information on program performance, but need to explain it to stakeholders and funders (legislative or private).

  • Tip – A complex evaluation can conduct multiple analyses to answer distinct questions and, thus, serve several purposes in a performance report. [Worker Assistance]
    • Tip – For oversight and funding decisions, legislators need descriptive information about the program, and information about intended and unintended effects, so they can understand the variety of conditions under which a program operates, the kind of coverage a targeted program has achieved, and how federal funds are spent. [Information to Congress].

A sound methodology can help identify effective programs and assess impact, but there is no guarantee clear answers will be obtained.  

  • Tip – An Evaluation study design can help examine effectiveness even where there are complex federal initiatives and where desired outcomes are infrequently observed [Designing Evaluations].
  • Tip – Even with well-conducted randomized experiments for assessing effectiveness when multiple causal influences create uncertainty about what caused results—study controls may not work, external factors may not have been assessed, or the results were not as expected. [Worker Assistance].

I hope I have identified some useful resources that are free, produced by experts, free of political bias, and have met the test of time. These are gems in education, justice, and human services.

Lesson Learned:

There is no one way to think about evaluation. GAO has focused on evaluation from the federal government perspective, but we can learn from one another and apply tools, methods, and concepts to the range of current challenges.

The American Evaluation Association is hosting GAO Week in celebration of the US Government Accountability Office’s 100th anniversary. The contributions all this week to aea365 come from authors who address GAO’s efforts to solve those complex “wicked” socio-cultural problems that defy permanent solutions but demand our best efforts to solve them. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

1 thought on “GAO Week: GAO’s 100th Celebration: Examples of How Agencies have Addressed Some Evaluation Challenges by Elaine Vaurio”

  1. Hello Ms. Vaurio,
    Firstly, congratulations on your long and prestigious career with GAO. I am a professional masters of education student at Queen university, currently taking a Program Inquiry Evaluation course, and I was drawn to this article because we are currently working through a module on program evaluation dilemmas and trying to create a program evaluation design, with this week’s task being to identify data collection methods and analysis strategies.

    I found your article useful in this regard, and enjoyed your straight forward tips and suggestions. The links to the reports that support your ideas were helpful in providing more explanation. I especially found GAO’s PROGRAM EVALUATION A Variety of Rigorous Methods Can Help Identify Effective Interventions, November 2009, offered some great ideas that support combining multiples measures or two or more designs in a study that would help obtain a more comprehensive picture of the effectiveness of the intervention. The article suggests alternative rigorous methods of assessing intervention effectiveness that go beyond randomized experiments, which you so kindly noted: quasi-experimental comparison groups, statistical analysis of observational data, in-depth case studies and collecting additional data and target comparisons to help reduce the influence of external factors and gather a diverse body of evidence.

    In my program design for my school’s student council, I am taking an outcome mapping approach to determine to what extent we are meeting our objectives to increase student body engagement and sense of belonging and to determine how program delivery can be improved. I hope to do what you suggested, “conduct multiple analyses to answer distinct questions and, thus, serve several purposes in a performance report”. I have never created a program evaluation design previously, so in this first attempt I am realizing just how complex a process it can be, and how challenging it is to determine a method of evaluation that will accurately measure effectiveness of a program and rule out competing causal explanations. Given your vast evaluation experience, what advice can you provide about how to best address the impact of external factors in evaluation results? What combination of methods for assessing effectiveness would you recommend in my program evaluation?
    Thanks for your time.

    Kind regards,
    Sarah Charlton

Leave a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.