Hi! I’m Brian Yates, co-chair of the Costs, Effectiveness, Benefits, and Economics TIG, founded in 2004. I supervise evaluation theses and dissertations at the Program Evaluation Research Lab (PERL) at American University in Washington DC, in our Clinical Psychology Ph.D. program.
Hot Tips:
“Cost-effective” doesn’t simply mean “cheap.” “Cost-beneficial” doesn’t simply mean “it’s worth it.” There’s so much more to cost-inclusive evaluation!
Resources are implicit in most theories of change and most program models. Resources and their monetary values (costs) can be made explicit in program models on the “front end” as shown in this figure. Increases in resources available for society (benefits) can be included too, as impacts beyond regular program outcomes.
Evaluations more often get used if they include information on the types, amounts, and monetary values of those ingredients that make programs possible, such as time, space, and telecommunications.
Adding information on improved income for participants, or reductions in their health care and other expenses beyond those of the program, can help advocate for program funding.
Many types of evaluation include costs, from cost feasibility assessments (CFA) and activity-based costing (ABC) through cost-effectiveness analysis (CEA), cost-benefit analysis (CBA), cost-utility analysis (CUA), and social return on investment (SROI). Each can range in complexity from simple to incredibly complex. Simpler usually is better understood and more likely to be used in decisions.
Rad Resources:
Here’s a short article that lays out the “banquet” of cost-inclusive evaluation methods available, with examples and more.
SROI is of increasing interest for many funders: here’s a special issue of an evaluation journal that explores SROI, its applications, and potential problems.
Cool Trick:
Often my students and I “piggy-back” our cost-inclusive evaluations on regular evaluations, hopefully adding value in exchange for analyzing data on costs, cost-effectiveness, and cost-benefit.
If an evaluation includes a program outcome, and if information is available on the amount of time spent by program implementers, basic cost-effectiveness analysis is possible. If outcomes include changes in use of health care, criminal justice, or other services, a cost-benefit analysis can be conducted.
Lessons Learned:
Like all evaluation, it is essential to include multiple stakeholders in as many steps of cost-inclusive evaluation as possible. For example, when evaluating mental health services, consumers as well as practitioners should be involved from the beginning: “Nothing about me without me”! Consumers as well as practitioners, also need to be asked about the resources they provide. Often consumers’ time spent in sessions and getting to and from clinics exceeds time practitioners spend directly with consumer and indirectly in supervision and case conferences.
Hi Brian,
I want to start by introducing myself, my name is Zach Unrau and I am currently in the Professional Master of Education program at Queen’s University where I am taking a course called Program Inquiry and Evaluation. In this course I have personally learned a lot about program evaluation however, we have not yet discussed much related to cost in evaluation which is why I wanted to take some time to review and reflect on your article. Some further information about the course is that for the course’s final project we are being asked to develop a program evaluation design and I have decided to create mine based off a program I run called the New Student Peer Mentoring Program (NSPMP).
In your article I really liked your point about using additional information about costs outside of the program to advocate for program funding however, I think this information could also apply to encouraging different types of program use as well. This for me was an important reminder that sharing evaluation results with different stakeholders requires different information to make decisions. Also while I do not think I will be able to incorporate cost feasibility assessments into my final project I do think that when considering next steps to take when implementing my final project into my work context I think I should start thinking about “piggy-backing” a cost inclusive evaluation onto the current evaluation model I have developed. Specifically, in the NSPMP one of the goals is to increase the amount of first-year students that continue past their first-year of studies to graduation and if we are successful being able to show a return on investment could help with program funding.
I am wondering though when adding cost-inclusive evaluations onto regular evaluations as an evaluator, what should I consider in order to make sure both evaluations run smoothly?
Best,
Zach
This is a great post! Thanks for sharing all the resources. At Partnerships For Health, we’ve used the CDC’s Learning and Growing Through Evaluation module on Economic Evaluation to plan our economic evaluations. It’s a great step-by-step guide!
https://www.cdc.gov/asthma/program_eval/asthmaprogramguide_mod6.pdf