Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the second of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.
This post presents the approach to the development of result chains and their integration within a Theory of Change (TOC) from a program perspective.
Step 2 – Developing the Program Theory of Intervention (PTI)
Program interventions are best modeled using chains of results with a program delivery (activity – output) sequence followed by an outcome sequence linking outputs to the program’s intended result (final outcome). Most models use only two levels of outcomes, although some authors advocate using as many as five. However, three levels of outcomes would seem to be optimal as it allows properly linking chains of results to broader TOCs, with the link being made through factors (immediate outcomes) that influence behaviors (intermediate outcomes) in target populations, in order to resolve the specific societal issue (final outcome) that has given rise to the program (see Figure 2a).
In chains of results, outputs are the products delivered by the program (as well as services, through a push-pull approach) that reach target populations, marking the transition between the sequence controlled by the program (i.e. program control zone) and the sequence controlled by recipients (i.e., influence zone of the program).
Logic models developed using this approach help clarify how the program intervention is assumed to achieve its intended results (i.e., the nested program theory of intervention) under the conditions defined in the broader TOC (see Figure 2b).
Developed this way, logic models do resolve a number of issues:
- The models provide a clear depiction of the chains of results and of the underlying working assumptions or hypotheses (i.e. salient causal links) of the program interventions and of their contribution to a common final result that is specific to the program;
- The models provide the basis to identify comprehensive sets of indicators supporting ongoing performance measurement (i.e. monitoring) and periodic evaluations, from which a subset can be selected for reporting purposes;
- Indicators can also cover external factor/risks that have (or may have) an ongoing influence on program results and that should be considered (i.e. included as control variables) in analyses to obtain more reliable assessments of program effectiveness.
However, developing a logic model that is a valid representation of program theories of interventions is easier said than done. The next AEA365 post will offer some suggestions for achieving that goal. Further, since logic models focus heavily on program outcomes, they provide very little information on delivery processes in support of management oversight and control. Subsequent posts will be discussing how program delivery can be meaningfully addressed and properly integrated in program theories of intervention.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
If you apply so simple program theory (top-down hierarchy, no feedback, no secondary impacts, no synergy) then your intervention logic is ok. However, governments usually operate with much more complex policy challenges, which require complex program theory – in such case, your intervention logic is not sufficient. My argument is further elaborated here (on the case of evaluating heritage policy impacts) https://www.sdeval.si/komisija-za-vrednotenje/publikacije/628-smart-heritage-policy-delovni-zvezek-let-x-st-1
I love this idea of chains of results. I wonder if you have seen any differences in how your clients react to this framing vs. the “traditional” logic model?
Most people like results chains and adopt them when they understand the way they can be used to articulate clear Program Theory of Intervention (PTI). However, they require much more discipline in developing logic model than what is required under the traditional approach, and some managers resist the greater transparency it provides when they find out their interventions are based on imprecise or ambiguous assumptions. Implementing strong approaches to performance measurement will always require a lot of proactive support from the consultant.
I agree, but the work is worthy and neccesary