AEA365 | A Tip-a-Day by and for Evaluators

TAG | Theory of Change

Hello, I am Carolyn Cohen, owner of Cohen Research & Evaluation, LLC, based in Seattle Washington. I specialize in program evaluation and strategic learning related to innovations in the social change and education arenas.  I have been infusing elements of Appreciative Inquiry into my work for many years.  Appreciative Inquiry is an asset-based approach, developed by David Cooperrider in the 1980s for use in organizational development. It is more recently applied in evaluation, following the release of Reframing Evaluation through Appreciative Inquiry by Hallie Preskill and Tessie Catsambas in 2006.

 Lessons Learned:

Appreciative Inquiry was originally conceived as a multi-stage process, often requiring a long-term time commitment. This comprehensive approach is called for in certain circumstances. However, in my practice I usually infuse discrete elements of Appreciative Inquiry on a smaller scale.  Following are two examples.

  • Launching a Theory of Change discussion. I preface Theory of Change conversations by leading clients through an abbreviated Appreciative Inquiry process.  This entails a combination of paired interviews and team meetings to:
    • identify peak work-related experiences
    • examine what contributed to those successes
    • categorize the resulting themes.

The experience primes participants to work as a team to study past experiences in  a safe and positive environment. They are then  able to craft  strategies, outcomes and goals. These elements become the cornerstone of developing a Theory of Change or a strategic plan, as well as an evaluation plan.

  • Conducting a needs assessment. Appreciative interviews followed by group discussions are a perfect approach for facilitating organization-wide or community meetings as part of a needs assessment process.   AI methods are  based on respectful  listening to each other’s stories, and are well-suited for situations where participants don’t know each other, or have little in common.

Using the resources listed below, you will find many more applications for Appreciative Inquiry in your work.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

We are Silvia Salinas-Mulder, Bolivian anthropologist, feminist activist and independent consultant, and Fabiola Amariles, Colombian economist, founder and director of Learning for Impact. We have worked for several years as external evaluators for development programs in Latin America. The following ideas may help to operationalize the principles of gender- and human rights (HR)-responsive evaluation in complex, multicultural contexts.

Lesson learned: Terms of Reference (TOR) for an evaluation are not engraved in stone.

Tip: Reframe the often conventional evaluation questions and other aspects of the evaluation process to ensure that gender and HR issues surface, and evidence of change (or no change) in women’s lives is gathered. Take into account context-specific issues and gender dynamics, as well as relevant cultural patterns, such as the effects of migration in the family roles and decision-making processes within some agricultural community settings.

Lesson learned: Some stakeholders are tired of being interviewed, while others – especially rural women- are eager to be heard.

Tip: Be creative; evaluation techniques are the means not the end, and can thus permanently be created, recreated and adapted to each situation and context. For example, use “conversatorios” (round table discussions), as opposed to focus groups, to gather people with diverse backgrounds and perspectives to discuss over a particular issue of the evaluation; participants usually appreciate these reflective spaces and feel motivated to speak “outside the box”, while evaluators take a holistic overview of the topic.  Drawings, role plays and other popular education techniques may also facilitate participation of marginalized groups, including illiterate women.

Lesson learned: Answers to your questions may not contain key gender and HR issues to understand how change is occurring.

Tip: Awareness of specific cultural and gender communication patterns is crucial for an effective exchange. In any case, interviews should be dealt with as dialogues where people have the opportunity to express their priorities and points of view. Do not limit your interactions to a question-answer dynamic. Let people speak freely and “listen actively” to discover the essential. Respect and interpret the silences and do not insist on answers to your questions, rather focus on trying to understand the underlying meaning of each reaction. This will allow an eventual reconstruction of how change is occurring (Theory of Change) for the specific intervention and context, even if it has not been explicitly stated in the project design. Also, as evaluators we tend to focus on verbal communication, ignoring the importance of tone and gestures. Make sure you are alert to less explicit key messages.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Chad Green here on the utility of Costa and Garmston’s maturing outcomes map from yesterday’s post.

If you revisit this colorful framework, you will notice that the nested concepts form a learning continuum, ranging from concrete to abstract, similar to the outcomes in a logic model.  However, if you dabble in cognitive neuroscience like myself, you may find this ordering of outcomes counterintuitive given the anatomical structure of the human brain. To correct for this minor oversight, all you need to do is invert the continuum such that the states of mind (i.e., empowerment outcomes) occupy the center of the model. According to my experience, this simple inversion of the continuum creates a “lensing” effect, if you will, that magnifies the valuing of the outcomes on the outer rings.

Hot Tip: Thinking in terms of nested outcome models has been very useful for my evaluation work.  For example, in order to make sense of my role as PreK-12 Program Co-chair, I created this earth metaphor as an adaptation of Alkin and Christie’s (2004) evaluation theory tree.  In theory, evaluators may span the layers of this conceptual framework in three ways: deductively, inductively, or transducively (both inside and outside).

  • Theories of change (Inner core): Provide evaluators with a central understanding of how growth and transformation occur within the evaluand across cultures, time, and space.
  • Values and principles in evaluation (Outer core): Determine the purpose of human valuing (i.e., of evidence, quality, inquiry, equity, social justice) within the context of the evaluator’s work.
  • Use for decision making and change (Mantle): Provide evaluators with roles, procedures, and perspectives to help users of evaluation information make decisions and instill change in more efficient or engaging ways.
  • Methods of evaluation research (Upper mantle): Serve as general practices in knowledge construction that emanate from and build upon the evaluator’s theories of change, values, and roles.
  • Domain knowledge (Crust): Specify subject matter expertise evaluators should possess for their line of work.

Rad Resources: Check out this video on the golden circle, a nested model developed by Simon Sinek.  Sinek’s framework has parallels in Universal Design for Learning, an approach to teaching that gives all individuals equal opportunities to learn.

Hot Tip: Consider this quote by Hegel (1817), author of The Science of Logic, in which he describes the nested nature of philosophy itself: “Each of the parts of philosophy is a philosophical whole, a circle rounded and complete in itself. … The single circle, because it is a real totality, bursts through the limits imposed by its special medium, and gives rise to a wider circle. The whole of philosophy in this way resembles a circle of circles.”

The American Evaluation Association is Educational Evaluation Week with our colleagues in the PreK-12 Educational Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EdEval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello, I am Glenn O’Neil and specialize in evaluating communication programs and campaigns with my own company, Owl RE. My post today is about how to use the theory of change in evaluating communication programs.

Hot tip: there is nothing so practical as a theory of change! The theory of change maps out from activities to impact how the communications action would bring about change, often in a flow-chart like diagram. Here is a simplified example:


This should be done when designing a communications action but in my experience it is rarely the case. So you can reconstruct the theory of change at the start of the evaluation – what activities were undertaken? What was the desired short and long term effects – for example, raising awareness amongst whom? Getting people to act, but on what? Mobilizing publics – but what for? This helps clarify what you are then going to measure and how to go about it.

Rad resource: For more examples of how the theory of change is used in campaign evaluation for non-profits, check out this excellent paper from Julia Coffman of Harvard University: “Lessons in evaluating communications campaigns: Five case studies. Harvard Family Research Project, 2002 (pdf)”.

For those that would like a broad overview of ‘how to’ evaluate communication programs and projects, check out my presentation slides for a “One day training workshop for communication professionals on evaluating communication programmes, products and campaigns”.

· · ·

<< Latest posts

Archives

To top