AEA365 | A Tip-a-Day by and for Evaluators

TAG | program theory

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the third of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post articulates how logic models should be structured when program designs include multiple strategic elements (or program activities) supporting a common objective.

Step 3 – Addressing Conditionality

Program interventions rarely rely on a single product or service to achieve intended results. In fact, program strategies are most often designed using multiple interventions from one or more players. In these situations, there normally exists some conditionality between separate program activities as they support and interact with each other. Addressed this way, the notion of conditions (also used in the TOC literature) allows structuring logic models by properly sequencing the multiple program interventions (i.e. converging results chains) deemed to contribute to a common final result that is specific to the program.

To an outside observer being exposed to multiple interventions, program activities may appear to be delivered in a sequential manner (from left to right) based on some observable results (e.g., outputs or immediate outcomes) until some final outcome is achieved (see Figure 3a). This would be the case of a person arriving at a hospital emergency or an employment centre and being subjected to a series of treatments or services.

However, from a program perspective, all activities are actually implemented in parallel with different clients and/or players. In the examples of the hospital emergency and the employment centre, it is the clients who are moving from left to right across activities as they are exposed to various program services. In programs that reach clients only indirectly (e.g., environmental programs or economic policies), it is rather the projects or client files that are shifting across activities while being processed and/or subjected to various program interventions.

Conditionality then allows taking into account the relationships between the strategic elements (or activities) of program interventions without the need to clutter the logic model with an exhaustive mapping and display of all possible interactions and feedback processes. Implicitly, all program activities are (or may be) influenced to some extent by previous activities situated at the left of the diagram (see Figure 3b). Thus, when conditionality exists and is properly taken into consideration, the positioning of program activities in the logic model becomes important for the description and understanding of the program theory of intervention (PTI).

The next AEA365 post will dwell further into program implementation and discuss how to best integrate delivery processes into logic models in order to effectively support management oversight and control.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the second of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post presents the approach to the development of result chains and their integration within a Theory of Change (TOC) from a program perspective.

Step 2 – Developing the Program Theory of Intervention (PTI)

Program interventions are best modeled using chains of results with a program delivery (activity – output) sequence followed by an outcome sequence linking outputs to the program’s intended result (final outcome). Most models use only two levels of outcomes, although some authors advocate using as many as five. However, three levels of outcomes would seem to be optimal as it allows properly linking chains of results to broader TOCs, with the link being made through factors (immediate outcomes) that influence behaviors (intermediate outcomes) in target populations, in order to resolve the specific societal issue (final outcome) that has given rise to the program (see Figure 2a).

 

In chains of results, outputs are the products delivered by the program (as well as services, through a push-pull approach) that reach target populations, marking the transition between the sequence controlled by the program (i.e. program control zone) and the sequence controlled by recipients (i.e., influence zone of the program).

Logic models developed using this approach help clarify how the program intervention is assumed to achieve its intended results (i.e., the nested program theory of intervention) under the conditions defined in the broader TOC (see Figure 2b).

Developed this way, logic models do resolve a number of issues:

  • The models provide a clear depiction of the chains of results and of the underlying working assumptions or hypotheses (i.e. salient causal links) of the program interventions and of their contribution to a common final result that is specific to the program;
  • The models provide the basis to identify comprehensive sets of indicators supporting ongoing performance measurement (i.e. monitoring) and periodic evaluations, from which a subset can be selected for reporting purposes;
  • Indicators can also cover external factor/risks that have (or may have) an ongoing influence on program results and that should be considered (i.e. included as control variables) in analyses to obtain more reliable assessments of program effectiveness.

However, developing a logic model that is a valid representation of program theories of interventions is easier said than done. The next AEA365 post will offer some suggestions for achieving that goal. Further, since logic models focus heavily on program outcomes, they provide very little information on delivery processes in support of management oversight and control. Subsequent posts will be discussing how program delivery can be meaningfully addressed and properly integrated in program theories of intervention.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

Hi! I’m Mary Arnold, a professor and 4-H youth development specialist at Oregon State University, where I spend the majority of my time in the area of program evaluation, especially in capacity building efforts. This is my second time preparing a blog post for the EEE-TIG, and the invitation came at a great time, because I have been thinking pretty obsessively these days on how we can do a better job of building Extension program planning and evaluation capacity. One of the conundrums and persistent late night ponderings that continues to rattle around my mind is how we can do a better job articulating what is suppose to take place in programs. If we are clear on of what is supposed to happen in a program, then we also should be able to predict certain outcomes and understand exactly how those outcomes come to be. This notion of prediction is what underscores a program’s theory.

Because of the emphasis on program planning and that swept Extension in the early 2000s, most Extension educators are familiar with logic modeling. The good news is that   many educators understand the concepts of inputs, outputs, and outcomes as a result, so the groundwork is in place to think more deliberately about a program’s theory. But at the same time, there is scant evidence that logic modeling has resulted in better program planning practices, or led to the achievement of stated outcomes in Extension programs. And there is even less evidence that logic models are developed based on theory.

Lesson Learned: Theory may be implied in logic models, but too often it is understated, assumed, or just hoped for. Program theory is what connects the components of a logic model and makes it run!

Hot Tip! Did you know that there are two important parts to program theory? The first is the program’s theory of change, which is the way in which the desired change comes about. The second is the program’s theory of action, which refers specifically to what actions need to happen, at what level of success, for the program to reach its intended outcomes.

Rad Resource! My favorite resource for understanding and developing a program theory of change and action is Purposeful program theory: Effective use of theories of change and logic models (Funnell & Rogers, 2011). This book has loads of great information and practical help on bringing logic models to life with program theory.

Rad Resource! If you are looking for specific theories that are useful for Extension programs, The University of Maryland Extension has a terrific short guide entitled Extension Education Theoretical Framework that outlines how several well-developed theories can be useful for Extension programming.

The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

I am Dawn Henderson, an Assistant Professor at Winston-Salem State University and former AEA GEDI (Graduate Education Diversity Initiative) intern. My experience as a GEDI afforded the opportunity to increase my knowledge in thinking about a program’s theory in evaluation. I used this in designing research and work with a school-based intervention. I want to share with you some tips I have used to help programs think about their particular theory and ways to evaluate and measure it. This is primarily targeted towards youth, so you can modify these tips for your specific needs.

Hot Tip: Conduct interviews with program staff and collect program materials. Interview key staff using questions like, What are unique characteristics of this program? How do these impact youth? When youth finish your program what do you want them to achieve or have? How would you measure that?  These perspectives can assist you with understand factors that may potentially lead to outcomes. Review program materials such as curriculum, lesson plans, and activities, and organize them into common themes. For example, I read through the lesson plans of one program and identified managing conflict as a consistent theme and used this to help develop program outcomes.

I also used the findings to identify literature to support the program’s efforts.  For example, I framed the activities of the program using research in positive youth development.

Hot Tip: Use visual aids.  Use a visual aid to draw connections between program characteristics and outcomes. This can be created using basic shape features in Microsoft Word or SmartArt.  I found that this visual aid helped the program think about how to achieve its objectives and communicate its model to external constituents.

Henderson 1

Hot Tip: The “if” and “then” logic. It appears that everyone talks about using logic models in evaluation. Using the information you collected from program staff, materials, and visual aid, develop a series of “if” and “then” statements. For example, IF program X provides activities in conflict management and resolution THEN youth participants will improve their ability to manage and resolve conflict.

Hot Tip: The evaluation plan. Here is when you start gearing the program and yourself up for the evaluation. This can be similar to logic model development; I integrate each of the previous steps in this phase and include objectives, measures, participants, analysis, and outcomes.  It not only helped me in thinking through the evaluation, but assisted the program in communicating their efforts to external constituents.

Henderson chart 3

Rad Resource: Program Theory and Logic Models by Amhert H. Wilder Foundation

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Apollo M Nkwake, a Research Associate Professor at Tulane University’s Disaster Resilience and Leadership Academy. I design monitoring and evaluation plans, conduct program evaluations and facilitate a graduate course in quantitative analysis of disaster resilience.

Lessons learned: An elaborate design that explicates assumptions on which programs are premised, is essential for viable program evaluation. In designing programs, stakeholders need to clearly articulate how the interventions they implement are supposed to eventually bring about the desired changes. Lack of clarity on how the change processes are expected to unfold masks critical risks to program success and eludes measurement of success.

Hot tip: An elaborate program theory needs to elaborate at least four kinds of assumptions:

  • Diagnostic Assumptions: stated as stakeholders’ expectations or beliefs of the major and minor causes of core problems.
  • Prescriptive assumptions: relate to the intervention or strategy devised for the problem or to reach a stated objective, which represents stakeholders’ beliefs of what could be the best ways to address the problem or need.
  • Transformational assumptions: While prescriptive assumptions are related to strategies (and alternatives) devised to address a problem, transformational assumptions relate to how the immediate results of a strategy program or intervention (outputs) are expected to lead to long term desired changes.
  • External assumptions: preconditions for program success that are beyond the control of program stakeholders.

Hot tip: Monitoring and evaluating program assumptions alongside program outputs, outcomes and research questions helps to increase understanding of a program’s success

Rad resource: Working with Assumptions in International Development Program Evaluation

Want to learn more from Apollo? Register for: Working with assumptions – key concepts and tools for program design, monitoring, and evaluation at Evaluation 2013 in Washington, DC!

This week, we’re featuring posts by people who will be presenting Professional Development workshops at Evaluation 2013 in Washington, DC. Click here for a complete listing of Professional Development workshops offered at Evaluation 2013. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Archives

To top