Steve Kymes on Incorporating Economic Evaluation into Program Evaluation

My name is Steve Kymes and I am the Director of the Center for Economic Evaluation in Medicine (CEEM) at Washington University in St. Louis. At CEEM, we assist investigators conducting clinical or community based research design, and implement and conduct cost-benefit and cost-effectiveness studies. While it may often seem that conducting valid and credible economic evaluation is simply a matter of basic arithmetic, it always requires thoughtful planning and is best done with the benefit of a multidisciplinary team.

Hot Tip #1: Make sure you know why you are doing the economic evaluation. In the language of economic evaluation we say that this is considering the “perspective of the analysis.” More pragmatically, you should always remember that the purpose of the program evaluation (and thus the economic component of it) is to influence one or more policy makers or stakeholders. That being the case, you need to be aware of what is important to that audience. Everything in the analysis—the definition of cost, how cost will be measured, the measures of effectiveness (or benefit), and the methods of estimation used are all determined by the perspective of the decision maker. So this should be the very first question you answer before you do anything else.

Hot Tip #2: As with so much else in program evaluation, your work will be much easier and the results will have increased validity if you involve people from the organization being evaluated. However, it is even more critical in this area of evaluation than with others. Assessment of cost data often takes the evaluator out of his/her area of expertise, and may also be beyond the expertise of the primary contacts at the organization as well. Therefore, it is essential to make early contact with the accounting and operations staff of the organization to learn how resources are tracked, costing methods used, and how data can be extracted.

Hot Tip #3: Experts in economic evaluation go by a number of descriptions–some are economists, some are outcomes researchers, and often, they have an MBA and a strong background in accounting or finance. Whoever they are, you will have a stronger result and avoid heading down many blind alleys if you involve an expert in your evaluation team from the very beginning. They will help you understand how to design the most efficient study, assist in interacting with the organization’s accounting team, and help you to communicate your results to policy makers.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

5 thoughts on “Steve Kymes on Incorporating Economic Evaluation into Program Evaluation”

  1. Thank you for this insightful post. As a health economist working in economic evaluation at the University of Birmingham, UK,I will only add to the tip #3 that experts in economic evaluation, at least in the UK, go as well as ‘health economists’.
    I would suggest researchers, previous to performing economic evaluations in the form of decision-analytic modelling (CEA, CUA) to check out guidelines for that particular purpose. See results of my recent systematic review of available guidelines in DAM in the following link: http://www.sciencedirect.com/science/article/pii/S1098301515000066

  2. The “bible” if you will for economic evaluation is a book by Michael Drummond et al titled “Methods of Economic Evaluation of Health Care Programmes” it is very readable and is up to date. I believe they published the second edition in 2005 and is available on Oxford University press. Last time I checked, Amazon also had several used copies. Mike Drummond has published a number of very readable papers on various aspects of the subject, as has Milt Weinstein out of Harvard. The journals Value in Medicine and Medical Decision making are great resources but can get a bit “techy”. I also have a paper that introduces the methods that came out in 2007 in the journal “Ophthalmic Epidemiology”. Thanks for the inquiry Deborah, best of luck as you continue your training.

  3. Thank you for your insightful post, Steve! As a current evaluation student at UCLA, I would love to expand my knowledge in the field of program evaluation. Do you recommend any literature introducing students to economic evaluation?

  4. Great recap of the steps involved with economic evaluation. My only clarification issue is with Hot Tip #3. “They have an MBA” is off the mark. Most in this area do have a master degree (typically in a behavioral science like economics) and understand finance, but – in my experience anyway – the MBA is not as common in the evaluation field.

Leave a Reply to Jon Vencil Cancel Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.