AEA365 | A Tip-a-Day by and for Evaluators

TAG | money

Hello again! I’m Brian Yates, your Treasurer, Director of the Program Evaluation Research Laboratory (PERL) at American University in Washington, DC, and a Professor there too.

A few weeks ago I wrote an AEA365 on “Doing Cost-Inclusive Evaluation. Part I: Measuring Costs.” Measuring the costs of programs is only part of cost-inclusive evaluation. This week we focusing on measuring the monetary outcomes of program. Here’s lessons, tips, and resources for your next evaluation.

Lesson Learned – Costs are not outcomes: BENEFITS are outcomes. Many is the time that I have heard seasoned evaluators and administrators say “program cost” when they meant “program benefit.” What I call the “outcome bias” prompts many people to see only what comes out of programs (outcomes), and not what goes into them (resources, measured as costs). In cost-inclusive evaluations, “benefits” mean “outcomes that are monetary, or that can be converted into monetary units, i.e., that are monetizable.”

Lesson Learned – Examples? Benefits that really “count” for many funders and consumers of human include: a) increased income to clients (and taxes paid to society) resulting from client participation in a program, plus b) savings to society resulting from decreased client use of costly health and other services, like emergency room visits and hospital stays.

Hot Tip – Convert effectiveness to benefits with simple multiplication. If you assure confidentiality before approaching a program’s clients, they’ll often tell you what health and other services they used in the past few months. Sample clients before, during, and after program participation to assess impacts of the program on clients’ use of other services. Validate with checks with those other services. Next, transform these impacts into monetary units: multiply a client’s frequency of service use by the cost of each service (average of health service provider’s fees for that service, for instance). Then, compare costs of services used before and after a program for clients, and you’ve measured a potential program benefit that speaks louder than other outcome measures: cost savings produced by the program!

Lesson Learned – Wow finding: programs often pay for themselves — several times over, and quickly! (Look for specifics on how to analyze these cost-benefit relationships in a future AEA365.)

Lesson Learned – Just ’cause it has a dollar sign in front of it doesn’t make the number “better.” Benefits (and costs) are no more valid that the data from whence they’re derived. The “GIGO” (Garbage In –> Garbage Out) principle works here: invalid benefit data can lead to big mistakes about program funding.

Resource: For examples of measuring benefits and combining them with costs for program funding recommendations, see: http://www.wsipp.wa.gov/auth.asp?authid=2

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hi! I’m Brian Yates, Professor in the Department of Psychology, and Director of the Program Evaluation Research Laboratory (PERL), at American University in Washington, DC. I’ve also been the AEA Treasurer for the past 3 years, and am looking forward to serving for 3 more.

I’ve included cost as well as outcome measures in my quantitative and qualitative evaluations since the mid-1970s.

Lesson Learned – 1) Costs are not money. Money’s just a way to get access to the resources that make programs work. What matters for programs, and what I measure when I’m evaluating costs, are people’s time — clients’ as well as staff’s, space used, transportation (of clients to and from programs, often) … and not just total time spent working in the program, but the amount of time spent in the different activities that, together, are the program.

Hot Tip: When asking stakeholders about program costs, I make a table listing the major activities of the program (therapy, groups, education, for example) in columns and the major resources used by the program (staff and client time, office space, transportation, for example) in rows. Different stakeholders put the amount of each resource that they use in each activity, and then compare others’ entries with their own. Insights into program operations often ensue!

Lesson Learned – 2) The most valuable resources may not have a price. Many programs rely on volunteered time and donated space and materials: these often don’t come with a monetary price attached. One can assign a monetary value to these resources according to what the same time from the same person would be paid in a job, but the most important thing to measure is the amount of time, the capabilities of the person, and ways they spent their time.

Lesson Learned – 3) When measured only as money, cost findings are instantly obsolete and do not aid replication. Inflation can quickly make specific monetary values for program costs out of date and, all too soon, laughably low. Translating 1980 dollars into 2011 dollars is possible, but still does not inform planners as to what specific resources are needed to replicate a program in another setting.

Lesson Learned – 4) When presenting costs, keep resources in their original units. Yes, time is money … but it comes in units of hours to begin with. Report both, and your audience will learn not just price but what it takes to make the program happen.

Rad Resource: Here’s a free on-line and down-loadable manual I wrote on formative evaluation of not only cost, but also cost-effectiveness and cost-benefit … and not just for substance abuse treatment! http://archives.drugabuse.gov/impcost/IMPCOSTIndex.html

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Archives

To top