AEA365 | A Tip-a-Day by and for Evaluators

Greetings, I am Frances Carter-Johnson, an Education Data Scientist responsible for evaluation, data analysis and building data analytic capacity within the National Science Foundation’s Division for Human Resource Development in the Directorate for Education and Human Resources.

To continue to move the Race and Class Dialogue forward, I want to take this opportunity to highlight alumni of AEA’s Graduate Education and Diversity Internship (GEDI) program who’s careers uniquely apply culturally responsive evaluation learned from the GEDI program. In celebration of the 15th Anniversary of the GEDI program, several AEA panels highlighted GEDI leaders, directors and alums. I’m excited to focus on unique links between the Race and Class Dialogue series and the current professional roles of a few GEDI alumni. Their experiences are ones that we can all consider as we all work to move the Race and Class dialogue and efforts forward in our professional and personal lives.

Hot Tip:

Culturally Responsive Evaluation (CRE) principles were applied throughout the Race and Class Dialogue and are critical to the evaluation community’s universal efforts to expand, improve and institutionalize evaluation into diverse organizations and environments.

Race and Class Lessons from the EVAL17 GEDI15 Alumni Panel:

  1. Tools, such as geospatial mapping, can be used to provide culturally responsive evidence for decreasing institutional and individual biases associated with race and class. Nichole Stewart, GEDI Alum from cohort 8 illustrated this in her talk, Mapping Equity in Educational Facilities Planning. Dr. Stewart, currently serves as Director of Facilities Planning for Baltimore City Public Schools where she uses a culturally responsive perspective to plan for public school facilities, including developing mapping tools to examine facility investments through an equity lens.
  1. Intentional focus on race and class is important to institutionalizing evaluation domestically as well as internationally. Neva Pemberton, GEDI Alum from cohort 6 and currently the International Chief of Education Planning for the Federation of St. Kitts and Nevis, discussed the successes and obstacles in building a culture of evaluation within the education sector in the Eastern Caribbean. Dr. Pemberton’s training in CRE, additional training with organizations such as UNESCO’s Institute of Education Planning, and intentionality towards relevant race and class issues for her environment are allowing her to influence education policies and practices aligned with United Nations Sustainable Development goals and to spread this influence to other nations.

Some of the GEDI15 Alumni Reflections Panel Participants (from Left to Right): Saul Maldonado, Brandi Gilbert, Jessica Johnson, Frances Carter-Johnson, Lisa Aponte-Soto, Chris St.Vil, Nichole Stewart, Alison Mendoza Walters

The American Evaluation Association is celebrating Multiethnic Issues in Evaluation (MIE) Week with our colleagues in the MIE Topical Interest Group. The contributions all this week to aea365 come from MIE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

We are Monique Liston, Ph.D. and Leah Peoples, Ph.D., and we are evaluators that work in Urban Education. We have a two part blog post dedicated to breaking down the debts owed to historically marginalized people.

The conversations on the “achievement gap” permeate critical dialogues about race in the field of education. This discourse limits our collective opportunity to address how internalized, institutional, ideological and interpersonal oppression have severed our deepest connections to one another — our ability to dignify each other as human beings. Our racialized identities marginalize us from one another, yet our pride – particularly within the MIE – allows us to see marginalization as an opportunity for resistance. Our goal is to liberate us all from how race, class, gender, and sexuality oppression stifles us, but we have to be in the position to share what we have learned to make our experiences a process of healing and restoration. Educators need to witness and examine this process as they develop their capacity to teach the next generation.  In our role as evaluators, we have the opportunity to disrupt the pervasive and persistent nature of oppression through evaluation design, practice, analysis, and presentation. Here are two lessons learned and one hot tip (shared in Part 2) that have helped us to do this urban educational contexts.

  1. Challenging achievement gap discourse. Gloria Ladson Billings (2006) gave us language to challenge achievement gap discourses. Achievement gap discourses problematize brown and black children as lacking or deficient before addressing the generational harm that racist, classist, sexist and homophobic institutions have caused. Ladson-Billings puts forth the “educational debt” as an opportunity for us to focus on what the institutions owe to children color as opposed to how children of color are unable to cope with the system. Evaluators have the power to change the dialogue in all aspects of the evaluation process.
  2. Recognizing quality education as a debt owed to marginalized and historically oppressed communities. With Ladson-Billings guidance, as individuals within the system we are constantly attempting to pay down a debt owed. The urgency, of which Martin Luther King, Jr. is famous for identifying is critical for educators to understand and apply within their classrooms, school buildings, and administrative districts. The system created a debt and our labor of love, time, and skill must serve as payment on that debt. Each moment in which we do not provide love, time, and skill that addresses the needs of those harmed by the system we are compounding interest — causing the debt to increase and preventing the young people owed from ever getting their justice — a balance paid in full. With this in mind, evaluators in urban education have an inherent goal with every evaluation. To what extent does this program, initiative, or opportunity pay down the debt owed to children of color?

Rad Resources:

Ladson-Billings, G. (2006). From the achievement gap to the education debt: Understanding achievement in U.S. schools. Educational Researcher, 35(7), 3–12.

The American Evaluation Association is celebrating Multiethnic Issues in Evaluation (MIE) Week with our colleagues in the MIE Topical Interest Group. The contributions all this week to aea365 come from MIE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Melvin Hall

Greetings, I am Melvin Hall, Professor of Educational Psychology at Northern Arizona University, a recent AEA Board Member, and moderator of the recently completed dialogue series. I am pleased to have been asked to kick off this week of posts from the MIE TIG.

Rad Resource #1: An Evaluation Voice from the Past

For years, I have read the AEA365 blog. I have often found information that fills a gap in my knowledge about how to get something done in evaluation, where I previously hadn’t a clue that the issue was even important. I would like to send you back to the evaluation profession’s past to a Rad Resource that is just as timely today as when originally penned by Lee Cronbach and Associates in 1980. Lee would undoubtedly smile if he knew “Toward the Reform of Program Evaluation” would be cited as a Rad Resource in 2017, but that is my contention. In our quest for the new and practical, the field has become estranged from important wisdom of the past.

The first of 95 thesis statements from that book says, “Program evaluation is a process by which society learns about itself”. This underscores the clarion call from the AEA Dialogues, that evaluation and evaluators take up more space in the public sphere where institutionalized sources of potential racism and classism should be identified and interrogated. The call is to have evaluative thinking become more prominent in public debate and policy reviews, an appeal to evaluators to identify and engage the important societal issues embedded in work we do. Who is better situated to engage those issues and promote societal reflection, dialogue, and change?

We know that inequity based upon gender is forcefully imposed by the powerful, who have bent institutions to their will, making them accomplices in hiding abuses and marginalizing victims. It is institutional power that silences victims and wipes away culpability; the same is done with racial animus and classist disdain.  Social institutions normalize this behavior, obscuring its damage, particularly from the perpetrators.

Rad Resource #2: Excuses, Excuses

This week consider how evaluators unwittingly play into the power of the status quo, and examine when and where our roles in this should end. Another Rad Resource–Racism without Racists by Eduardo Bonilla-Silva–exposes the excuses we often use to explain why the institutions we serve are not perpetuating inequity, or if they are, why it is not our place to intervene.

Lesson Learned: In the end, society will benefit if we use our skills to hold up a mirror so that it can learn about its collective self. Through posts this week the MIE TIG will highlight relevant lessons learned and examined through a variety of lenses. The online discussion forum for the Dialogues is open and available, if this week of posts prompts you to want to join the discussion.

The American Evaluation Association is celebrating Multiethnic Issues in Evaluation (MIE) Week with our colleagues in the MIE Topical Interest Group. The contributions all this week to aea365 come from MIE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the last of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This last post discusses the creation and analysis of comprehensive and integrated databases for ongoing performance measurement (i.e., monitoring), periodic evaluation and reporting purposes.

Step 7 – Collecting and Analysing Data

Program interventions are usually designed to be delivered in standardized ways to target populations. However, standardization does not often take into consideration variations in circumstances that may affect the results of interventions, such as:

  • Individual differences (e.g., demographic and psychological factors);
  • Contextual variables (e.g., social, economic and geo-political factors/risks);
  • Program and institutional variables (e.g., type and level of services, delivery method, accessibility).

Focusing exclusively on program delivery (i.e., economy and efficiency) through the assessment of the achievement of delivery targets or compliance with delivery standards may be quite appropriate when programs are mature, and the causal relationships between outputs and outcomes are well understood and well established. But this is not always the case, and definitely not so when programs are new or in a demonstration phase (e.g., pilot project) and relying on uncertain or unverified underlying assumptions. In those situations, more robust and adapted analytical techniques should be used to measure the extent to which programs interventions actually contribute to observed results while taking external factors in to account. This is essential to the reliable assessment of program impacts/outcomes.

It is well known in econometrics that incomplete explanatory models lead to biased estimators because the variance that should have been taken by missing variables is automatically redistributed among the retained explanatory variables. Translated for evaluation, this means that excluding external factors from analytics creates a risk of incorrectly crediting the program with some levels of impacts that should instead have been attributed to the missing variables (i.e., having the program claim undue responsibility for observed results).

Dealing with this issue would require collecting appropriate microdata and creating complete data sets, holding information on all explanatory variables for each member of target populations, which can then be used to:

  • Conduct robust multivariate analysis to isolate the influence of program variables (i.e., reliably assessing program effectiveness and cost-effectiveness) while taking all other factors into account;
  • Explore in a limited way (using the resulting regressive model to extrapolate) how adjustments or tailoring of program delivery to specific circumstances could improve program outcomes;
  • Empirically assess delivery standards as predictive indicators of program outcomes (rather than rely exclusively on benchmarking) to determine requisite adjustments to existing program delivery processes.

Developing successful program interventions will require the evaluation function to successfully deal with the above challenges and more effectively support management decision-making processes.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the sixth of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post discusses how to identify comprehensive sets of indicators supporting ongoing performance measurement (i.e. monitoring) and periodic evaluations, from which subsets of indicators can be selected for reporting purposes.

Step 6 – Defining Performance Indicators

When a logic model is developed using the structured approach presented in previous posts, and gets validated by management (and stakeholders), it can be deemed to be an adequate description of the Program Theory of Intervention (PTI). The task of identifying performance indicators then requires determining a comprehensive set of indicators that includes some reliable measure(s) of performance for each output, outcome and external factor covered by the logic model. In some cases, as discussed in the previous post, the set of indicators may extend to cover management issues as well.

Most performance measurement strategies (PMS) and scorecards also require the identification, for each output and outcome, of success criteria such as performance targets and/or standards, which are usually based on some form of benchmarking. This is consistent with a program design mode (i.e. top-down approach to logic models) based on inductive logic where each result is assumed to be a necessary and/or sufficient condition (as discussed in the TOC literature) for achieving the next level of results. This is however very limiting as it reduces the discussion of program improvement and/or success to the exclusive examination of performance in program delivery (as proposed in Deliverology).

Additional useful information that may be required includes the following:

  • Data type (quantitative or qualitative);
  • Data source (source of information for data collection);
  • Frequency of data collection (e.g. ongoing, tied with specific events, or at fixed intervals);
  • Data owner (organization responsible for data collection);
  • Methodology (any addition al information about measurement techniques, transformative calculations, baselines and variable definitions that must be taken into consideration in selecting analytical techniques);
  • Scales (and thresholds) used for assessing and visually presenting performance;
  • Follow-up or corrective actions that should be undertaken based on performance assessments.

Many organizations further require that performance measurement be designed with a view to address evaluation needs and adequately support the periodic evaluation of the relevance, efficiency and effectiveness of program interventions. However, evaluation and performance measurement strategies are most often designed separately, with evaluation strategies usually being arrested just before the actual conduct of evaluation studies. Evaluations are then constrained by the data collected and made available through performance measurement. In order for evaluation and performance measurement strategies to be coordinated and properly integrated, it would be necessary to develop them concomitantly at an early stage of program implementation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the fifth of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post discusses how the relation of performance measurement to results-based management should be articulated and incorporated into logic models.

Step 5 – Including the Management Cycle

Some logic models try to include management as a program activity leading to corporate results (e.g., ‘financial/operational sustainability’ and ‘protection of organization’) that are presented as program outcomes. Indeed, good management can help improve program delivery and thus contribute to program performance. However, that contribution is indirect and normally achieved through the ongoing oversight and control of program delivery (and the occasional revision of program design) with requisite adjustments to operational or strategic plans being informed by the ongoing measurement (or monitoring) and the periodic assessments of program performance (see Figure 5a).

Results-based management (RBM) then depends on the identification of relevant indicators and the availability of valid and reliable data to correctly inform players/stakeholders and adequately support management reporting and decision-making processes. The quality and use of performance measurement systems for governance is actually one of many elements of Management Accountability Frameworks (MAF) in the Canadian Federal Government, with other elements covering expectations regarding stewardship, policy and program development, risk management, citizen-focused service, accountability and people management. However, MAFs are the object of development and an assessment process that is totally separate from the one used for Performance Measurement Frameworks (PMF) based on delivery process models and/or logic models.

Indeed, the management cycle is relatively independent from actual program operations, with management standing in a relation of authority above program staff to provide oversight and control at each step of the delivery process (see Figure 4b in yesterday’s AEA365 post).

Trying to build the management cycle as a chain of results (or as a part of it) in a logic model is then totally inappropriate as it creates unnecessary confusion between management and program performance issues. Presenting the results of good management as program outcomes also blurs the distinction between efficiency (i.e., the internal capacity to deliver) and effectiveness (i.e., program impacts on target populations). Figure 5b below shows how to properly situate the management cycle in a logic model itself, essentially as an authoritative or facilitative process without direct causal links to specific program results.

This does not mean that management issues should be excluded from PMFs. Relevant indicators of management performance should also be identified for monitoring purposes whenever they are identified by management itself as internal factors or risks that do (or may) influence program delivery.

The next AEA365 post will discuss ways of addressing indicators and actual measures of performance.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the fourth of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post articulates the approach to the development of delivery process models that ventilate the individual activity-output sequences of a logic model in order to allow for the management oversight and control of program implementation.

Step 4 – Including Delivery Processes

Delivery process modelling can easily be accomplished by adapting a computer-assisted Integrated Definition (IDEF) methods developed in the ‘90s by the National Institute of Standards and Technologies (NIST). The IDEF 0 function modelling was initially based on authority links of controls ensuring that activities of multiple players are coordinated and undertaken only when proper authorities have been issued. In the IDEF 0 model, inputs and support mechanisms are used to perform operations and produce outputs through operations or transformation processes that are subjected to management controls and oversight (see Figure 4a). These nodes can be detailed further (by digging into the steps of each function) and/or sequenced to provide an exhaustive view of all program operations.

The delivery process model presented in Figure 4b is an adaptation of the IDEF 0 approach that is achieved by redefining operations and mechanisms as successive sub-activities supported by various players and stakeholders, with distinct products that are delivered at each step of production and that are subject to direct management authority (i.e. oversight and control). The final step is then the one that actually generates the product (or service) that is being identified as the output of a specific activity in the logic model.

In this modelling approach, inputs are not used to define consumed resources (e.g. funds and human resources), but rather to identify the mechanisms and sources of support for program delivery. Further, since delivery is done in a strict stepwise manner, the strong conditionality between sub-activities can actually be redefined as dependencies. The model also makes it possible to take into account internal and external factors/risks that can have an influence on delivery at each step of production.

The above model is essentially a working tool to help analysts validate their understanding of delivery processes. In performance measurement frameworks, the description of the process models can actually be limited to the sequence of operations and their related products in the narrative of each activity of the logic model, with a view to support the identification of indicators for monitoring purposes.

Figure 4b already suggests how to position the management cycle in relation with program delivery processes, and how best to articulate management issues from a program perspective. The next AEA365 post will address these features in greater detail.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the third of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post articulates how logic models should be structured when program designs include multiple strategic elements (or program activities) supporting a common objective.

Step 3 – Addressing Conditionality

Program interventions rarely rely on a single product or service to achieve intended results. In fact, program strategies are most often designed using multiple interventions from one or more players. In these situations, there normally exists some conditionality between separate program activities as they support and interact with each other. Addressed this way, the notion of conditions (also used in the TOC literature) allows structuring logic models by properly sequencing the multiple program interventions (i.e. converging results chains) deemed to contribute to a common final result that is specific to the program.

To an outside observer being exposed to multiple interventions, program activities may appear to be delivered in a sequential manner (from left to right) based on some observable results (e.g., outputs or immediate outcomes) until some final outcome is achieved (see Figure 3a). This would be the case of a person arriving at a hospital emergency or an employment centre and being subjected to a series of treatments or services.

However, from a program perspective, all activities are actually implemented in parallel with different clients and/or players. In the examples of the hospital emergency and the employment centre, it is the clients who are moving from left to right across activities as they are exposed to various program services. In programs that reach clients only indirectly (e.g., environmental programs or economic policies), it is rather the projects or client files that are shifting across activities while being processed and/or subjected to various program interventions.

Conditionality then allows taking into account the relationships between the strategic elements (or activities) of program interventions without the need to clutter the logic model with an exhaustive mapping and display of all possible interactions and feedback processes. Implicitly, all program activities are (or may be) influenced to some extent by previous activities situated at the left of the diagram (see Figure 3b). Thus, when conditionality exists and is properly taken into consideration, the positioning of program activities in the logic model becomes important for the description and understanding of the program theory of intervention (PTI).

The next AEA365 post will dwell further into program implementation and discuss how to best integrate delivery processes into logic models in order to effectively support management oversight and control.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the second of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies as well as broader initiatives involving multiple organizations.

This post presents the approach to the development of result chains and their integration within a Theory of Change (TOC) from a program perspective.

Step 2 – Developing the Program Theory of Intervention (PTI)

Program interventions are best modeled using chains of results with a program delivery (activity – output) sequence followed by an outcome sequence linking outputs to the program’s intended result (final outcome). Most models use only two levels of outcomes, although some authors advocate using as many as five. However, three levels of outcomes would seem to be optimal as it allows properly linking chains of results to broader TOCs, with the link being made through factors (immediate outcomes) that influence behaviors (intermediate outcomes) in target populations, in order to resolve the specific societal issue (final outcome) that has given rise to the program (see Figure 2a).

 

In chains of results, outputs are the products delivered by the program (as well as services, through a push-pull approach) that reach target populations, marking the transition between the sequence controlled by the program (i.e. program control zone) and the sequence controlled by recipients (i.e., influence zone of the program).

Logic models developed using this approach help clarify how the program intervention is assumed to achieve its intended results (i.e., the nested program theory of intervention) under the conditions defined in the broader TOC (see Figure 2b).

Developed this way, logic models do resolve a number of issues:

  • The models provide a clear depiction of the chains of results and of the underlying working assumptions or hypotheses (i.e. salient causal links) of the program interventions and of their contribution to a common final result that is specific to the program;
  • The models provide the basis to identify comprehensive sets of indicators supporting ongoing performance measurement (i.e. monitoring) and periodic evaluations, from which a subset can be selected for reporting purposes;
  • Indicators can also cover external factor/risks that have (or may have) an ongoing influence on program results and that should be considered (i.e. included as control variables) in analyses to obtain more reliable assessments of program effectiveness.

However, developing a logic model that is a valid representation of program theories of interventions is easier said than done. The next AEA365 post will offer some suggestions for achieving that goal. Further, since logic models focus heavily on program outcomes, they provide very little information on delivery processes in support of management oversight and control. Subsequent posts will be discussing how program delivery can be meaningfully addressed and properly integrated in program theories of intervention.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

AEA365 Curator note: Today begins a special theme week with an extended (7 day) series on one topic by one contributing author. 

Hello, my name is Michel Laurendeau, and I am a consultant wishing to share over 40 years of experience in policy development, performance measurement and evaluation of public programs. This is the first of seven (7) consecutive AEA365 posts discussing a stepwise approach to integrating performance measurement and evaluation strategies in order to more effectively support results-based management (RBM). In all post discussions, ‘program’ is meant to comprise government policies and broader initiatives involving multiple organizations.

Step 1 of 7 – Developing the Theory of Change (TOC)

Effectively addressing an issue normally requires first understanding what you are dealing with. Models are generally used in evaluation to help clarify how program are meant to work and achieve intended results. However, much confusion exists between alternative approaches to modelling, each based on different ways of representing programs and the multiple underlying assumptions on which their interventions are based.

Top-down models, such as the one presented in Figure 1a, usually provide a narrow management perspective relying on inductive logic in order to select the evidence (based on existing knowledge and/or beliefs) that is necessary to support ex ante the strategic and operational planning of program interventions. Assumptions are then entirely about whether the program created necessary and/or sufficient conditions (as discussed in the TOC literature) for achieving intended results. In this context, the role of ex post evaluation is too often limited to focusing on program delivery and vindicating management’s contention that observed results depend to some (usually unknown) extent on existing program interventions.

As a research function, evaluation should also support (re)allocation decisions being made by senior government officials regarding the actual funding of public programs. However, this stronger evaluation role would involve reliably assessing individual program contributions to observed results in a given context, and require properly measuring real/actual program impacts while taking external factors into account.

The first difficulty in achieving this task is recognizing that Randomized Control Trials (RCT) are rarely able to completely eliminate the influence of all external factors, and that the statistical ‘black box’ approach it uses prevents reliably transposing (i.e., forecasting by extrapolating) observed results to situations with varying circumstances. Generalization is then limited to a narrow set of conditions formulated as broad assumptions about the context in which the program operates. Providing a more extensive base to reliably measure program effectiveness would entail, in a first step:

  1. developing more exhaustive Theories of Change (TOC) including all factors that created the need for program interventions and/or that likely have an influence on the issue or situation being addressed by the program; and,
  2. determining which factors/risks within the TOC are meant to be explicitly ‘managed’ by the program, with all others becoming external to the program intervention.

Figure 1b shows what a program logic model would normally look like at the end of this first step.

The next AEA365 post will articulate the approach to the development of the more detailed Program Theory of Intervention (PTI) that is imbedded within the broader TOC.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

<< Latest posts

Older posts >>

Archives

To top