AEA365 | A Tip-a-Day by and for Evaluators

TAG | results-based management

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the last of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This last post discusses the creation and analysis of comprehensive and integrated databases for ongoing performance measurement (i.e., monitoring), periodic evaluation and reporting purposes.

Step 7 – Collecting and Analysing Data

Program interventions are usually designed to be delivered in standardized ways to target populations. However, standardization does not often take into consideration variations in circumstances that may affect the results of interventions, such as:

  • Individual differences (e.g., demographic and psychological factors);
  • Contextual variables (e.g., social, economic and geo-political factors/risks);
  • Program and institutional variables (e.g., type and level of services, delivery method, accessibility).

Focusing exclusively on program delivery (i.e., economy and efficiency) through the assessment of the achievement of delivery targets or compliance with delivery standards may be quite appropriate when programs are mature, and the causal relationships between outputs and outcomes are well understood and well established. But this is not always the case, and definitely not so when programs are new or in a demonstration phase (e.g., pilot project) and relying on uncertain or unverified underlying assumptions. In those situations, more robust and adapted analytical techniques should be used to measure the extent to which programs interventions actually contribute to observed results while taking external factors in to account. This is essential to the reliable assessment of program impacts/outcomes.

It is well known in econometrics that incomplete explanatory models lead to biased estimators because the variance that should have been taken by missing variables is automatically redistributed among the retained explanatory variables. Translated for evaluation, this means that excluding external factors from analytics creates a risk of incorrectly crediting the program with some levels of impacts that should instead have been attributed to the missing variables (i.e., having the program claim undue responsibility for observed results).

Dealing with this issue would require collecting appropriate microdata and creating complete data sets, holding information on all explanatory variables for each member of target populations, which can then be used to:

  • Conduct robust multivariate analysis to isolate the influence of program variables (i.e., reliably assessing program effectiveness and cost-effectiveness) while taking all other factors into account;
  • Explore in a limited way (using the resulting regressive model to extrapolate) how adjustments or tailoring of program delivery to specific circumstances could improve program outcomes;
  • Empirically assess delivery standards as predictive indicators of program outcomes (rather than rely exclusively on benchmarking) to determine requisite adjustments to existing program delivery processes.

Developing successful program interventions will require the evaluation function to successfully deal with the above challenges and more effectively support management decision-making processes.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the sixth of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post discusses how to identify comprehensive sets of indicators supporting ongoing performance measurement (i.e. monitoring) and periodic evaluations, from which subsets of indicators can be selected for reporting purposes.

Step 6 – Defining Performance Indicators

When a logic model is developed using the structured approach presented in previous posts, and gets validated by management (and stakeholders), it can be deemed to be an adequate description of the Program Theory of Intervention (PTI). The task of identifying performance indicators then requires determining a comprehensive set of indicators that includes some reliable measure(s) of performance for each output, outcome and external factor covered by the logic model. In some cases, as discussed in the previous post, the set of indicators may extend to cover management issues as well.

Most performance measurement strategies (PMS) and scorecards also require the identification, for each output and outcome, of success criteria such as performance targets and/or standards, which are usually based on some form of benchmarking. This is consistent with a program design mode (i.e. top-down approach to logic models) based on inductive logic where each result is assumed to be a necessary and/or sufficient condition (as discussed in the TOC literature) for achieving the next level of results. This is however very limiting as it reduces the discussion of program improvement and/or success to the exclusive examination of performance in program delivery (as proposed in Deliverology).

Additional useful information that may be required includes the following:

  • Data type (quantitative or qualitative);
  • Data source (source of information for data collection);
  • Frequency of data collection (e.g. ongoing, tied with specific events, or at fixed intervals);
  • Data owner (organization responsible for data collection);
  • Methodology (any addition al information about measurement techniques, transformative calculations, baselines and variable definitions that must be taken into consideration in selecting analytical techniques);
  • Scales (and thresholds) used for assessing and visually presenting performance;
  • Follow-up or corrective actions that should be undertaken based on performance assessments.

Many organizations further require that performance measurement be designed with a view to address evaluation needs and adequately support the periodic evaluation of the relevance, efficiency and effectiveness of program interventions. However, evaluation and performance measurement strategies are most often designed separately, with evaluation strategies usually being arrested just before the actual conduct of evaluation studies. Evaluations are then constrained by the data collected and made available through performance measurement. In order for evaluation and performance measurement strategies to be coordinated and properly integrated, it would be necessary to develop them concomitantly at an early stage of program implementation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the fifth of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post discusses how the relation of performance measurement to results-based management should be articulated and incorporated into logic models.

Step 5 – Including the Management Cycle

Some logic models try to include management as a program activity leading to corporate results (e.g., ‘financial/operational sustainability’ and ‘protection of organization’) that are presented as program outcomes. Indeed, good management can help improve program delivery and thus contribute to program performance. However, that contribution is indirect and normally achieved through the ongoing oversight and control of program delivery (and the occasional revision of program design) with requisite adjustments to operational or strategic plans being informed by the ongoing measurement (or monitoring) and the periodic assessments of program performance (see Figure 5a).

Results-based management (RBM) then depends on the identification of relevant indicators and the availability of valid and reliable data to correctly inform players/stakeholders and adequately support management reporting and decision-making processes. The quality and use of performance measurement systems for governance is actually one of many elements of Management Accountability Frameworks (MAF) in the Canadian Federal Government, with other elements covering expectations regarding stewardship, policy and program development, risk management, citizen-focused service, accountability and people management. However, MAFs are the object of development and an assessment process that is totally separate from the one used for Performance Measurement Frameworks (PMF) based on delivery process models and/or logic models.

Indeed, the management cycle is relatively independent from actual program operations, with management standing in a relation of authority above program staff to provide oversight and control at each step of the delivery process (see Figure 4b in yesterday’s AEA365 post).

Trying to build the management cycle as a chain of results (or as a part of it) in a logic model is then totally inappropriate as it creates unnecessary confusion between management and program performance issues. Presenting the results of good management as program outcomes also blurs the distinction between efficiency (i.e., the internal capacity to deliver) and effectiveness (i.e., program impacts on target populations). Figure 5b below shows how to properly situate the management cycle in a logic model itself, essentially as an authoritative or facilitative process without direct causal links to specific program results.

This does not mean that management issues should be excluded from PMFs. Relevant indicators of management performance should also be identified for monitoring purposes whenever they are identified by management itself as internal factors or risks that do (or may) influence program delivery.

The next AEA365 post will discuss ways of addressing indicators and actual measures of performance.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the fourth of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post articulates the approach to the development of delivery process models that ventilate the individual activity-output sequences of a logic model in order to allow for the management oversight and control of program implementation.

Step 4 – Including Delivery Processes

Delivery process modelling can easily be accomplished by adapting a computer-assisted Integrated Definition (IDEF) methods developed in the ‘90s by the National Institute of Standards and Technologies (NIST). The IDEF 0 function modelling was initially based on authority links of controls ensuring that activities of multiple players are coordinated and undertaken only when proper authorities have been issued. In the IDEF 0 model, inputs and support mechanisms are used to perform operations and produce outputs through operations or transformation processes that are subjected to management controls and oversight (see Figure 4a). These nodes can be detailed further (by digging into the steps of each function) and/or sequenced to provide an exhaustive view of all program operations.

The delivery process model presented in Figure 4b is an adaptation of the IDEF 0 approach that is achieved by redefining operations and mechanisms as successive sub-activities supported by various players and stakeholders, with distinct products that are delivered at each step of production and that are subject to direct management authority (i.e. oversight and control). The final step is then the one that actually generates the product (or service) that is being identified as the output of a specific activity in the logic model.

In this modelling approach, inputs are not used to define consumed resources (e.g. funds and human resources), but rather to identify the mechanisms and sources of support for program delivery. Further, since delivery is done in a strict stepwise manner, the strong conditionality between sub-activities can actually be redefined as dependencies. The model also makes it possible to take into account internal and external factors/risks that can have an influence on delivery at each step of production.

The above model is essentially a working tool to help analysts validate their understanding of delivery processes. In performance measurement frameworks, the description of the process models can actually be limited to the sequence of operations and their related products in the narrative of each activity of the logic model, with a view to support the identification of indicators for monitoring purposes.

Figure 4b already suggests how to position the management cycle in relation with program delivery processes, and how best to articulate management issues from a program perspective. The next AEA365 post will address these features in greater detail.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the third of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post articulates how logic models should be structured when program designs include multiple strategic elements (or program activities) supporting a common objective.

Step 3 – Addressing Conditionality

Program interventions rarely rely on a single product or service to achieve intended results. In fact, program strategies are most often designed using multiple interventions from one or more players. In these situations, there normally exists some conditionality between separate program activities as they support and interact with each other. Addressed this way, the notion of conditions (also used in the TOC literature) allows structuring logic models by properly sequencing the multiple program interventions (i.e. converging results chains) deemed to contribute to a common final result that is specific to the program.

To an outside observer being exposed to multiple interventions, program activities may appear to be delivered in a sequential manner (from left to right) based on some observable results (e.g., outputs or immediate outcomes) until some final outcome is achieved (see Figure 3a). This would be the case of a person arriving at a hospital emergency or an employment centre and being subjected to a series of treatments or services.

However, from a program perspective, all activities are actually implemented in parallel with different clients and/or players. In the examples of the hospital emergency and the employment centre, it is the clients who are moving from left to right across activities as they are exposed to various program services. In programs that reach clients only indirectly (e.g., environmental programs or economic policies), it is rather the projects or client files that are shifting across activities while being processed and/or subjected to various program interventions.

Conditionality then allows taking into account the relationships between the strategic elements (or activities) of program interventions without the need to clutter the logic model with an exhaustive mapping and display of all possible interactions and feedback processes. Implicitly, all program activities are (or may be) influenced to some extent by previous activities situated at the left of the diagram (see Figure 3b). Thus, when conditionality exists and is properly taken into consideration, the positioning of program activities in the logic model becomes important for the description and understanding of the program theory of intervention (PTI).

The next AEA365 post will dwell further into program implementation and discuss how to best integrate delivery processes into logic models in order to effectively support management oversight and control.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the second of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post presents the approach to the development of result chains and their integration within a Theory of Change (TOC) from a program perspective.

Step 2 – Developing the Program Theory of Intervention (PTI)

Program interventions are best modeled using chains of results with a program delivery (activity – output) sequence followed by an outcome sequence linking outputs to the program’s intended result (final outcome). Most models use only two levels of outcomes, although some authors advocate using as many as five. However, three levels of outcomes would seem to be optimal as it allows properly linking chains of results to broader TOCs, with the link being made through factors (immediate outcomes) that influence behaviors (intermediate outcomes) in target populations, in order to resolve the specific societal issue (final outcome) that has given rise to the program (see Figure 2a).

 

In chains of results, outputs are the products delivered by the program (as well as services, through a push-pull approach) that reach target populations, marking the transition between the sequence controlled by the program (i.e. program control zone) and the sequence controlled by recipients (i.e., influence zone of the program).

Logic models developed using this approach help clarify how the program intervention is assumed to achieve its intended results (i.e., the nested program theory of intervention) under the conditions defined in the broader TOC (see Figure 2b).

Developed this way, logic models do resolve a number of issues:

  • The models provide a clear depiction of the chains of results and of the underlying working assumptions or hypotheses (i.e. salient causal links) of the program interventions and of their contribution to a common final result that is specific to the program;
  • The models provide the basis to identify comprehensive sets of indicators supporting ongoing performance measurement (i.e. monitoring) and periodic evaluations, from which a subset can be selected for reporting purposes;
  • Indicators can also cover external factor/risks that have (or may have) an ongoing influence on program results and that should be considered (i.e. included as control variables) in analyses to obtain more reliable assessments of program effectiveness.

However, developing a logic model that is a valid representation of program theories of interventions is easier said than done. The next AEA365 post will offer some suggestions for achieving that goal. Further, since logic models focus heavily on program outcomes, they provide very little information on delivery processes in support of management oversight and control. Subsequent posts will be discussing how program delivery can be meaningfully addressed and properly integrated in program theories of intervention.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

AEA365 Curator note: Today begins a special theme week with an extended (7 day) series on one topic by one contributing author. 

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the first of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies and broader initiatives involving multiple organizations.

Step 1 of 7 – Developing the Theory of Change (TOC)

Effectively addressing an issue normally requires first understanding what you are dealing with. Models are generally used in evaluation to help clarify how program are meant to work and achieve intended results. However, much confusion exists between alternative approaches to modelling, each based on different ways of representing programs and the multiple underlying assumptions on which their interventions are based.

Top-down models, such as the one presented in Figure 1a, usually provide a narrow management perspective relying on inductive logic in order to select the evidence (based on existing knowledge and/or beliefs) that is necessary to support ex ante the strategic and operational planning of program interventions. Assumptions are then entirely about whether the program created necessary and/or sufficient conditions (as discussed in the TOC literature) for achieving intended results. In this context, the role of ex post evaluation is too often limited to focusing on program delivery and vindicating management’s contention that observed results depend to some (usually unknown) extent on existing program interventions.

As a research function, evaluation should also support (re)allocation decisions being made by senior government officials regarding the actual funding of public programs. However, this stronger evaluation role would involve reliably assessing individual program contributions to observed results in a given context, and require properly measuring real/actual program impacts while taking external factors into account.

The first difficulty in achieving this task is recognizing that Randomized Control Trials (RCT) are rarely able to completely eliminate the influence of all external factors, and that the statistical ‘black box’ approach it uses prevents reliably transposing (i.e., forecasting by extrapolating) observed results to situations with varying circumstances. Generalization is then limited to a narrow set of conditions formulated as broad assumptions about the context in which the program operates. Providing a more extensive base to reliably measure program effectiveness would entail, in a first step:

  1. developing more exhaustive Theories of Change (TOC) including all factors that created the need for program interventions and/or that likely have an influence on the issue or situation being addressed by the program; and,
  2. determining which factors/risks within the TOC are meant to be explicitly ‘managed’ by the program, with all others becoming external to the program intervention.

Figure 1b shows what a program logic model would normally look like at the end of this first step.

The next AEA365 post will articulate the approach to the development of the more detailed Program Theory of Intervention (PTI) that is imbedded within the broader TOC.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello! My name is Michele Tarsilla, an independent evaluation advisor and evaluation capacity development (ECD) specialist with experience in over thirty countries. I have served as Chair of the International and Cross-Cultural Evaluation TIG at AEA and am currently in transition to become the new OL-ECB TIG Co-Chair.

The idea for this blog sprang from the realization that decision- and policy-makers participating in a number of ECD initiatives around the world are not always cognizant of the differences between evaluation (whose purposes and methods have often been exposed to only for a couple of years) and Results-Based Management (RBM) which they have been taught for almost two decades. As a result, the evaluation logic is often associated with the practice of developing logical frameworks (a phenomenon I refer to as “RBM-ization of the evaluation function”) and the potential of cross-pollination between both fields has not been fully capitalised. In an effort to fill the knowledge gap, I developed a table on links between evaluation and RBM (see a snapshot of the actual tool I developed for the OECD/DAC Newsletter and check out Rad Resource #1).

RBM vs. Evaluation (Tarsilla, 2014)

Tarsilla 1

Tarsilla 2

While I used to stress the intrinsic diversity between evaluation and RBM (e.g., by presenting evaluation as a radically new and more extensive endeavor than RBM has ever been), I came to change my position on this topic over the last year.

Lesson Learned: I have come to terms with the fact that, for an ECD programme to be designed and implemented successfully, the language used in evaluation workshops, mentoring programmes and technical assistance sessions needs to build upon the terminology, processes and tools (e.g., RBM-related) which clients and partners in the field are already familiar with. This renewed “ECD opportunism”, has proved to be quite effective in my professional practice. Trainees and clients exposed to this approach appear to have assimilated, retained and used evaluation concepts and tools more effectively.

Hot Tip #1: I strongly suggest recognition of the purposes (and related concept and tools) of RBM and performance management every time you are developing evaluation workshop curricula and designing evaluation technical assistance programmes, especially those aimed at planning and management specialists.

Hot Tip #2: It is critical to ensure that evaluation methodologies and concepts disseminated as part of an ECD programme fit within the realm of leaders’ existing results-driven management practices. This new strategy, which I call the “normalization of the evaluation function”, appears to help avoid the risk of rejection and apathy.

Rad Resources:

  • For a more exhaustive review of contemporary ECD practices in international development, visit http://www.oecd.org/dac/evaluation/ecdnewsletter.htm
  • For a an in-depth analysis of a selected number of ECD-related issues, visit http://www.ecdg.net/projects-partnerships/ecd-global-scan/ecd-weekly-blog-series/
  • For a review of international ECD initatives funded by international funders in Africa, see Tarsilla, M. (2014). Evaluation capacity development in Africa: Current landscape of international partners’ initiatives, lessons learned and the way forward. African Journal of Evaluation, 2, 34-58

The American Evaluation Association is celebrating Organizational Learning and Evaluation Capacity Building (OL-ECB) Topical Interest Group Week. The contributions all this week to aea365 come from our OL-ECB TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Archives

To top