My name is Jose Diaz, and I am one of the economists in Wilder Research conducting social return on investment analysis.
How many times have you been approached by a program leader asking you for an evaluation and just after a few minutes of talking you realize that the program does not have enough data for you to conduct the evaluation? You know that from this point on, it’s a very slippery road; data collection takes time and is expensive, and the already small budget for evaluation that the program had just became tiny and insufficient to pay for the research. So, is it all lost or is there an alternative?
Our team faces scenarios like this one very often. In these cases we suggest to our clients a type of analysis we call prospective social return on investment. This type of studies use secondary data to reasonably argue that the program in question is achieving its outcomes; we can then compute economic benefits and costs from the effect sizes and other existing data. This is not a completely new idea; the Washington State Institute for Public Policy has been a pioneer using meta-analysis to conduct economic analysis.
Hot Tips: Here are some considerations when conducting prospective return on investment:
- Defining the logic model of the intervention during the preliminary stages of the analysis would significantly help the process.
- Not all outcomes can be monetized; but maybe not having the outcome may cause some costs to society.
- We always need to keep in mind what is the alternative we are measuring the outcome against; that is, what would be the characteristics and conditions of the target population in the absence of the program? Usually, the target population in the studies found in the literature reviews is not the same as the target population served by the program we are evaluating; so we need to define and describe the population to which we can apply the effect sizes found.
- Along the way we will need to make assumptions that may be crucial in the final results; being conservative and reasonable about these assumption is a always a good practice
Lesson Learned: The idea sounds good in theory but in practice the devil is in the details. Social return on investment (prospective or actual) is an exercise in persuasion as much as a research effort, so you need to be your own worst enemy. Be mindful of the meaning and scope of the estimations produced. If you won’t, someone else will.
The American Evaluation Association is celebrating with our colleagues from Wilder Research this week. Wilder is a leading research and evaluation firm based in St. Paul, MN, a twin city for AEA’s Annual Conference, Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.