AEA365 | A Tip-a-Day by and for Evaluators

TAG | cost analysis

My name is Kelly Smith, and I’m an evaluator, policy analyst and economist at ECONorthwest, a consulting firm in Portland, Oregon. I’ve been a devoted member of OPEN (Oregon Program Evaluators Network), an AEA affiliate, since I was introduced to it in graduate school in 2007. OPEN works hard to provide our members with a variety of learning opportunities, including brown bag lunch talks, workshops, networking events, conferences, and a book club. We do this mostly by tapping into our own members’ expertise. Until recently, I’d been instrumental in planning these events, and I’d certainly been an enthusiastic participant, but I hadn’t yet felt comfortable leading one.

This past year, I overcame my reticence and agreed to give a talk about cost analysis, a topic that seems to raise fear and confusion, if not hackles, among many evaluators. The event was free and the room was packed, far exceeding our expectations. Evidently, people were hungry for knowledge about this subject. Some attendees even expressed surprise that we didn’t charge for such a valuable class (lesson learned!). Having sensed a large untapped demand, we decided to offer two sessions about cost analysis at our annual conference in the spring, and both were well attended. We got great feedback about the usefulness of this topic.

These sessions were my first experiences “teaching”, and I found it both intellectually challenging and valuable (and if I’m honest, a bit nerve-wracking). I had to study the topic to refresh my knowledge, practice public speaking and presentation skills, and think on my feet. I’m confident that I got more out the experience than the attendees.

Get involved! If you’re a member of your local affiliate, don’t hesitate to step up and contribute your own knowledge! There is a real demand for learning opportunities, and certainly not enough supply. Your colleagues will benefit from your willingness to share, your affiliate will become stronger and more active, and you’ll come out ahead, too.

It’s hard to understate the value of learning from each other as evaluators. Not only do we grow as professionals when we contribute to our colleagues and our affiliate, but we expand the value and reach of the evaluation profession as a whole.

Rad Resource: If you are interested in learning more about cost analysis in evaluation, you’ll find a copy of the slides used at the conference sessions here.

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings all! I am Agata Jose-Ivanina and I am a Senior Associate with ICF International, a research and evaluation firm headquartered near Washington, DC. As the budget debate has been heating up and educational agencies have to make hard choices, we have seen an increased demand to include a cost component as part of program evaluation. When trying to measure and analyze costs, evaluators should follow four steps:

In Step 1, you need to identify all of the possible costs of a program. Using the “ingredients method,” list out all of the possible cost categories of a program, as well as the individual costs within each category. For example, an evaluation of an educational technology program might include costs for personnel, hardware, software, training, troubleshooting, and maintenance.

Hot Tip: When identifying cost ingredients, it is a good idea to work together with the program staff who know the most about the program’s operations.

In Step 2, you develop a calculation for each of the individual costs identified in Step 1. For example, to estimate the value of the time that teachers spent in training, you would multiply the length of the training by the number of teachers by the hourly stipend that teachers receive for participating in professional development.

Hot tip: In this step, write out a formula for each calculation. Writing out all formulae will help you understand what data you need to collect and all the assumptions you may need to make.

In Step 3, you collect each of the individual pieces of data that is necessary for Step 2 calculations. Evaluators can’t rely solely on program budgets when collecting cost data.  Budgets often bundle costs of several programs together, and it may be hard to isolate the information that you need. Moreover, budgets may not include items that were not explicitly paid for, such as the cost of office space or time from salaried employees.

Hot Tip: Receipts, interviews, market research, and surveys — a researcher may need to employ all these strategies to get at the total cost of the program.

Step 4: Once you have your data, it is just a matter of plugging them into the calculations you developed in Step 2, and adding them up to get a total program cost.

Rad Resource: If you are new to cost-benefit or cost-effectiveness analysis, the book to start with is Levin and McEwan’s Cost-Effectiveness Analysis: Methods and Applications. Not only does it explain the concepts very well, it has a wonderful bibliography.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Agata? She’ll be presenting as part of the Evaluation 2011 Conference Program, November 2-5 in Anaheim, California.

My name is Steve Kymes and I am the Director of the Center for Economic Evaluation in Medicine (CEEM) at Washington University in St. Louis. At CEEM, we assist investigators conducting clinical or community based research design, and implement and conduct cost-benefit and cost-effectiveness studies. While it may often seem that conducting valid and credible economic evaluation is simply a matter of basic arithmetic, it always requires thoughtful planning and is best done with the benefit of a multidisciplinary team.

Hot Tip #1: Make sure you know why you are doing the economic evaluation. In the language of economic evaluation we say that this is considering the “perspective of the analysis.” More pragmatically, you should always remember that the purpose of the program evaluation (and thus the economic component of it) is to influence one or more policy makers or stakeholders. That being the case, you need to be aware of what is important to that audience. Everything in the analysis—the definition of cost, how cost will be measured, the measures of effectiveness (or benefit), and the methods of estimation used are all determined by the perspective of the decision maker. So this should be the very first question you answer before you do anything else.

Hot Tip #2: As with so much else in program evaluation, your work will be much easier and the results will have increased validity if you involve people from the organization being evaluated. However, it is even more critical in this area of evaluation than with others. Assessment of cost data often takes the evaluator out of his/her area of expertise, and may also be beyond the expertise of the primary contacts at the organization as well. Therefore, it is essential to make early contact with the accounting and operations staff of the organization to learn how resources are tracked, costing methods used, and how data can be extracted.

Hot Tip #3: Experts in economic evaluation go by a number of descriptions–some are economists, some are outcomes researchers, and often, they have an MBA and a strong background in accounting or finance. Whoever they are, you will have a stronger result and avoid heading down many blind alleys if you involve an expert in your evaluation team from the very beginning. They will help you understand how to design the most efficient study, assist in interacting with the organization’s accounting team, and help you to communicate your results to policy makers.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

My name is Kathryn Hill and I work with the state of Minnesota’s GEAR UP college access initiative as the Evaluation and Research Manager. GEAR UP is the acronym for the Gaining Early Awareness and Readiness for Undergraduate Programs federal grant program, designed to increase the number of low-income students who are prepared to enter and succeed in postsecondary education. GEAR UP provides six-year grants to states and partnerships to provide services at high-poverty middle and high schools.

I collaborated with external evaluators to develop a framework for a cost-benefit study. I learned a lot from this evaluation management experience, and I have compiled a few tips.

Hot Tip- Start with a thorough literature review: To develop a cost analysis framework, you need evidence to support proposed outcomes and to attribute economic value for these proposed outcomes. It is possible to rely on existing research to determine a projected “effect” for the program when building a framework, but you should have results from your own rigorous evaluation before proceeding with the cost study.

Hot Tip- Use evidence to articulate the program theory: A logic model is a common starting point for communicating program theory, and it is essential for cost analysis.Using program theory, an evaluator can move toward a clear identification of the different program components. These program components guide the economic calculations of “inputs”. Financial calculations of program expenses are viewed from program staff and participant perspectives. Both are important for determining the cost of a program. If it takes a lot of time for a staff member to develop/deliver a program component that is utilized by only a small number of students, the cost per participant will be high for that specific component. This brings us back to the “effect” issue, because you may want to know what proportion can be attributed to each of your program components.

Hot Tip- Find a way to document/describe what is actually happening in the program: The external evaluators developed an interactive format for interviewing all program staff. Staff members found the process interesting; some even thought it was fun!

Hot Tip- Think “program components” rather than “accounting categories” when recording expenses: For example, our program provides college visits, and the cost includes transportation expenses. However, that budget category usually has transportation expenses for EVERYTHING, including field trips, summer programs, etc. You will save yourself many headaches if you set up detailed sub-codes for expense records.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio, to learn more from Kathryn.

Archives

To top