Hello! My name is Jacqueline Singh, MPP, PhD, Executive Evaluation & Program Design Advisor at Qualitative Advantage, LLC. I’m a long-time member of Indiana Evaluation Association. I served as IEA Director at Large, Policy Committee Chair and continue to contribute to the IEA Program Committee.
Throughout my career I’ve held positions as an evaluator, researcher, and policy analyst. As a result, I’ve observed how program implementors and decision makers are incentivized to advance well designed programs and impactful policies. In addition, government leaders and constituents ask disparate types of questions that often inform decisions and policymaking. When considering overarching evaluation questions, it’s important to recognize that some questions are actually return on investment (ROI) questions. Unlike others, these questions require evidence to show that costs associated with an intervention are worthwhile and taxpayers’ dollars made a difference. Given the array of approaches to plan an evaluation, how might one proceed to answer a return on investment question?
For this AEA365, I briefly describe a well-recognized approach that I’ve coupled with program theory and evaluability assessment—and, other evaluative approaches and methods. It is known as the Phillips ROI Methodology®. Initially developed for business, it has evolved into an enhanced logic model that is comprised of 12 logical and systematic steps and guided by 12 standards. In addition, it integrates design thinking principles into four process stages useful for all types of projects and programs funded by government, NGO’s, as well as nonprofits. Overall, it can be described as a process that cuts across organizational boundaries and links initiatives, programs, projects and processes to bottom line measures.
In a nutshell, the Phillips ROI Methodology® enhanced logic model captures varying levels of impact and categorizes evaluation data. There are five levels and each has a measurement focus, that collectively capture evidence needed to answer questions regarding an intervention’s success:
Level 1 collects data to answer questions such as: a) is the program relevant to participants’ needs? b) did the program provide new information—or, raise awareness? c) are participants willing to recommend the program to others?
Level 2 measurements collect data about participants’ knowledge and/or level of confidence to apply newly acquired knowledge and skills after the program.
Level 3 captures changes in individuals’ behavior, as well as specific actions taken (e.g., in the workplace, community, organization, business, etc.). Level 3 is a rich source of data that can help answer questions such as: a) to what extent did participants apply newly acquired knowledge and skills? b) how successful are/were participants with their application? c) what supports are in place (or, not) to assist individuals in applying their knowledge and skills?
Level 4 measures change(s) that link an intervention with business or organizational impact (e.g., productivity, quality, cost, time, etc.). This type of measurement addresses the consequence of applying skills and knowledge from the program to answer questions such as: a) what is the business connection to the program? b) what impacts in the organization are attributable to the program?
Lastly, Level 5 compares monetary benefit to the costs associated with an intervention/program and calculates the return on investment (ROI). The ROI calculation is a standard financial equation developed through finance and economics:
BCR = Program Benefits
Program Costs
ROI (%) = Net Program Benefits
Program Costs
Rad Resources
A versatile reference on ROI methodology is: Value for Money: How to Show the Value for Money for All Types of Projects and Programs in Governments, Nongovernmental Organizations, Nonprofits, and Business. In addition, a plethora of free tools are available on the ROI Institute website.
The American Evaluation Association is hosting Indiana Evaluation Association (IEA) Affiliate Week. The contributions all this week to AEA365 come from IEA members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.