AEA365 | A Tip-a-Day by and for Evaluators

TAG | principles-focused evaluation

Hi there! I am Marah Moore, the founder and director of i2i Institute (Inquiry to Insight). We are based in the high desert mountains of Northern New Mexico, and we work on evaluations of complex systems locally, nationally, and internationally.

Since 2008 I have been the lead evaluator for the McKnight Foundation’s Collaborative Crop Research Program (CCRP), working in nine countries in Africa and three countries in the Andes. In 2014 the CCRP Leadership Team (LT), guided by the evaluation work, began an intentional process of identifying principles for the program. Up to that point we had developed a robust and dynamic theory of change (ToC) that guided program evaluation, learning, planning, and implementation. The ToC helped bring coherence to a complex and wide-ranging program. Because we wanted the ToC to remain a living document, growing and changing as the program grew and changed, we found we needed to identify a different sort of touchstone for the program—something that would anchor the conceptual and practical work of the program without inhibiting the emergence that is at the core of CCRP. That’s when we developed principles.

CCRP has eight overarching principles. The principles guide all decision-making and implementation for the program, and inform the development of conceptual frameworks and evaluation tools.

In addition to the principles at the program level, we have developed principles for various aspects of the program.

Lesson Learned: Programs based on principles expect evaluation to also be principles-based. Here are the draft principles we are using for the CCRP Integrated Monitoring & Evaluation Process.

  1. Make M&E utilization-focused and developmental
  2. Ensure that M&E is informed by human systems dynamics and the adaptive cycle: What? So what? Now what?
  3. Design M&E to serve learning, adaptation, and accountability
  4. Use multiple and mixed methods.
  5. Embed M&E so that it’s everyone’s responsibility
  6. Align evaluation with the Theory of Change.
  7. Ensure that M&E is systematic and integrated across CCRP levels
  8. Build M&E into project and program structures and use data generated with projects and programs as the foundation for M&E.
  9. Aggregate and synthesize learning across projects and time to identify patterns and generate lessons.
  10. Communicate and process evaluation findings to support ongoing program development and meet accountability demands.
  11. Ensure that evaluation follows the evaluation profession’s Joint Committee Standards.

Hot Tip: The evaluation process can surface principles of an initiative, exposing underlying tensions and building coherence. The evaluation can go further and assess the “fidelity” of an initiative against the principles and explore the role of the principles in achieving outcomes. 

Rad Resources:

The American Evaluation Association is celebrating Principles-Focused Evaluation (PFE) week. All posts this week are contributed by practitioners of a PFE approach. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· · ·

Hi, I’m Nora F. Murphy, co-founder of TerraLuna Collaborative. I fell in love with the approach when I first practiced it in 2012 as a mother first, and then as an evaluator.

In “Principles-Focused Evaluation: The Guide”, I share what it was like to make sense of life after my four-year-old son died. Reflecting on my parenting, it was clear that rules didn’t matter—not the number of vegetables he ate, screen minutes he viewed, or vocabulary words he heard. So what did matter? I loved him fiercely. I have no doubt he knew that. I valued his imagination, his laugh, his questions, the spark in his eye. I valued him and what he brought to our family and the world. When I could, I kept him safe. But I couldn’t keep him and when I couldn’t, I walked with him when he was scared. My son knew he was loved, he was valued, and that I would walk with him when he was scared. That’s what truly mattered. These have become my personal guiding principles.

As I made my way from grieving back to life, I wondered: What is my purpose? Is it possible to find and fulfill my purpose through evaluation? If so, how? Can I reimagine myself as an evaluator that works in alignment with who I am becoming, not who I was? Waking Lumina, my guiding principles for professional engagement emerged. These are:

  1. Engage heart, mind, and spirit in all aspects of living my life: my relationship with myself, my relationship with others, my work, and the decisions I make.
  2. Make choices that let my light shine more brightly, and engage with others in a way that supports their ability to shine more brightly.
  3. Build and deepen connections between and amongst people, spirit, nature, passion and purpose.
  4. Increase social justice and equity, recognizing my privilege and the opportunities it affords me to create change.
  5. Inspire and be inspired.

Since 2012, I have conducted numerous principles-focused developmental evaluations for social-justice oriented systems change. I engage my Waking Lumina principles to guide how I approach the work, and I help people in complex systems discover and define their own guiding principles. People I’m working are ecstatic, relieved, curious, or all of the above when I describe a principles-focused approach. They find that principles allow us to work together while seeing the world as it actually is, see people as they are, bring people together around hard issues without asking for complete agreement of uniformity, and provide a framework for coherent systems change with room for adaptation. It’s the most human way of practicing evaluation I’ve ever experienced.

Rad Resource: Read more about the first principles-focused developmental evaluation from the 2014 AEA365 DE Week post, Homeless Youth Collaborative on Developmental Evaluation.

Hot Tip: Follow the Waking Lumina blog to learn more about how the Waking Lumina guiding principles play out in life and evaluation: www.wakinglumina.com

Red Resource:  Principles-Focused Evaluation: The GUIDE. 

The American Evaluation Association is celebrating Principles-Focused Evaluation (PFE) week. All posts this week are contributed by practitioners of a PFE approach. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Charmagne E. Campbell-Patton and I am the Director of Organizational Learning and Evaluation at Utilization-Focused Evaluation. Over the last two years, I’ve had the privilege of facilitating the development of principles with a number of different groups. While the processes we used and the principles that emerged were distinct in each case, all of the groups struggled with the question of how much to include. Most wanted way more principles than made sense lest something critically important be left out.

Lesson Learned: Principles are most useful when they can be easily called to memory by everyone who is responsible for them. That usually means somewhere between four to eight principles, though of course there is no hard and fast rule. The general principle in this case is Less is More: Have enough principles to be meaningful and effective, but not so many that no one can remember (much less implement) them.

This is where the distinction between overarching and operating principles can come in handy.

Hot Tip: Overarching principles provide general guidance for effectiveness. Operating principles provide more specific guidance for implementing the overarching principles. Operating principles should still follow the GUIDE criteria.

Cool Trick: When working with a group to develop principles, let the ideas flow freely before trying to sort out which ones are overarching and which are operating. Once you have a full list, try to sort them into categories and determine which are most inspirational or core, and which are more tactical or supporting of the overarching idea.

Lesson Learned: Separating principles into overarching and operating principles helps to ensure that as many voices as possible are included without ending up with an unworkable number of principles.

Rad Resource: Chapter 17 in Principles-Focused Evaluation (Patton 2017) provides on example of the distinction between overarching and operating principles.

The American Evaluation Association is celebrating Principles-Focused Evaluation (PFE) week. All posts this week are contributed by practitioners of a PFE approach. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Ricardo Wilson-Grau, an independent evaluator based in Rio de Janeiro who over the last 15 years developed, with colleagues, the Outcome Harvesting approach to monitoring and evaluating development and social change interventions. I devote a fourth of my time to sharing and learning, including supporting these and other colleagues to use Outcome Harvesting.

As an approach to evaluate interventions that operate in complex, dynamic and diverse environments, this utilization-focused and highly participatory approach requires that the six steps of Outcome Harvesting be customized for each evaluation design and adapted in the course of the evaluation process. Suffice to say that in over fifty evaluations, I have never applied Outcome Harvesting the same way.

Consequently, I have found in my own practice and in that of colleagues that the need to mold and adapt to different client needs and contexts, tends to undermine fidelity to the principles underlying each of the six methodological steps —you can modify or skip one or more the steps but only if you maintain the principles. For example, you can define outcomes in different ways just so long as they are understood as observable, demonstrated changes in behavior. Similarly, an outcome can only be considered a product of an intervention if you can establish a plausible cause-effect contribution, but not attribution. A third example: substantiation is unnecessary if the original data is credible enough for the intended uses. That is, faulty application of Outcome Harvesting is due mainly to evaluators misunderstanding, misusing, or simply not taking into account principles such as these. And, success in using Outcome Harvesting generally flows from being true to the principles, in my experience at least.

Lesson Learned: The solution I am pursuing is to emphasize Outcome Harvesting as a principles-focused evaluation approach, flushing out and promoting the use of ten principles that are the methodological foundation of the approach.

Rad Resources: For Outcome Harvesting resources, examples of applications to monitoring and evaluation, and a calendar of events, visit this website I maintain to influence the quality and support fellow practitioners. To compare and contrast Outcome Harvesting with other utilization-focused, participatory and complexity-aware approaches to evaluation, visit Better Evaluation.

Hot Tip: In addition to visiting the website and reading Chapter 32 about Outcome Harvesting in Michael Quinn Patton’s book Principles-Focused Evaluation — The Guide, participate in the Outcome Harvesting Forum that is influencing fidelity to the principles and forging a community of practice.

The American Evaluation Association is celebrating Principles-Focused Evaluation (PFE) week. All posts this week are contributed by practitioners of a PFE approach. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hola! My name is Pablo Vidueira. I am an evaluation researcher and consultant working with the Global Alliance for the Future of Food (GA), a collaboration of philanthropic foundations that have come together to strategically push the agenda for more sustainable food systems globally.

But how can a group of more than 20 philanthropic foundations, with their different backgrounds, perspectives, priorities and agendas come together and define what to do and how to do it, within such an entangled and multifaceted system as the global food system? Here is where principles come to play.

Hot Tip #1: Principles provide a common ground for innovation

The Alliance members came together around the sense of unsustainability of the current food systems, but the question was how to go about addressing those problems. There is not a roadmap for such and endeavour, so how would you start to define what the GA should be and do? By setting goals? Strategies? Priorities? Projects?

For the Alliance, principles allowed to explicitly define a shared purpose from where strategies, goals, working groups and initiatives emerged.

The GA principles are listed below. These are general enough to provide common ground while specific enough to provide shared direction. These are collaboration principles, and they did not conflict with any such statements by the individual foundations.

 https://futureoffood.org/about-us/how-we-work/

Hot Tip #2: The convening power of principles

Innovation-related contexts greatly benefit from a shared framework for understanding and action as the GUIDE principles allow to build. The aforementioned principles are greatly internalized by the Alliance and the members. Anyone can clearly explain the principles and their relevance. Principles are used to determine whether an action should be supported by the GA, and in some cases they also allow to reframe the initiative. Anything supported by the Global Alliance needs to address each and every of the GA principles.

The following GA website snapshot shows their vision, which is built upon the principles.

https://futureoffood.org/about-us/how-we-work/

Hot Tip #3: Becoming a principles-driven alliance led to ongoing principles-focused evaluation

The Global Alliance adopted a Developmental Evaluation approach to help them evolve and adapt in their aim to leverage food systems towards a greater sustainability. As the principles emerged along this process, the Alliance became increasingly principles-driven, thus M&E at the Alliance needed to adapt and respond by becoming increasingly principles-focused. Thereby, PFE allows to examine the meaningfulness of the principles to the Alliance and their members, the adherence to the principles in the variety of initiatives sponsored by the Alliance, and the results of adhering to the principles. This is now a key pillar of all the Monitoring and Evaluation at the Global Alliance.

 Rad Resources:

  1. More information about the Global Alliance on their website. You can also subscribe to the newsletter or follow them on twitter: @futureoffoodorg
  1. Chapter 28 of the PFE book further analyses the case of the Global Alliance.
  2. The GUIDE framework also provided in the PFE book is a great tool to determine whether principles provide meaningful guidance (G) and are useful (U), inspiring (I), developmentally adaptable (D), and evaluable (E).

The American Evaluation Association is celebrating Principles-Focused Evaluation (PFE) week. All posts this week are contributed by practitioners of a PFE approach. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Michael Quinn Patton, author of Principles-Focused Evaluation: The GUIDE.  Principles-based initiatives, including programs, projects, collaborations, and change efforts of all kinds, base what they do and why they do it on guiding principles. Examples:

  • Build mutually respectful relationships
  • Engage in trauma-informed care
  • Act collaboratively

Principles inform and guide decisions and choices.  Principles-driven people are motivated by deeply help values expressed through principles that translate values into behaviors. Principles-focused evaluation makes principles the focus of evaluation (the evaluand).  Principles-focused evaluation examines (1) whether principles are clear, meaningful, and actionable, and if so, (2) whether they are actually being followed and, if so, (3) whether they are leading to desired results.  Principles are derived from experience, expertise, values, and research.

The GUIDE framework is a set of criteria for clarifying effectiveness principles, that is, a tool to help a principles-based program ensure readiness for principles-focused evaluation. GUIDE is an acronym and mnemonic specifying the criteria for a high quality principle statement.  A high quality principle (1) provides guidance, (2) is useful, (3) inspires, (4) supports ongoing development and adaptation, and (5) is evaluable.

GUIDE Framework for Effectiveness Principles

Essentially, GUIDE criteria provide an evaluability assessment framework applied to principles. 

Lessons Learned:  Evaluating principles is different from evaluating projects and programs.

Hot Tip:  Engaging in principles-focused evaluation often involves helping clarify principles to articulate them in a way that can be evaluated.

Cool Trick: Turn values (beliefs) into principles by adding an imperative verb that makes the values actionable and evaluable

Value statement: We believe in social justice.

Principle: Design interventions to support social justice.

Lesson Learned: Effectiveness principles are especially useful for navigating the turbulence and uncertainties of complex dynamic systems.

Rad Resource: Principles-Focused Evaluation: The GUIDE

The American Evaluation Association is celebrating Principles-Focused Evaluation (PFE) week. All posts this week are contributed by practitioners of a PFE approach. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Archives

To top