Hey everyone! I’m Greta Landis (she / her), an Evaluation Specialist based in Teejop (Madison, Wisconsin) at the Natural Resources Institute at the University of Wisconsin-Madison, and Program Co-Chair of the Environmental Program Evaluation TIG.
My work is informed by Dr. Michael Quinn Patton’s ‘principles-focused evaluation’ approach, which provides a set of strategies to develop guiding, values-based statements to use in the evaluation of complex, adaptive, mission-driven programs. Our evaluation team at Extension and colleague Courtney Bolinson at Head & Heart Evaluation have used this approach with a growing number of grant-funded, transdisciplinary agroecosystems research groups in the USDA’s Sustainable Agriculture Systems Program for the last four years. We have found that working with project teams to identify their principles provides a structured space to reflect on project terminology and communication, discuss meaningful partnerships, and consider their group processes, power dynamics, and plans.
However, measuring the adherence to, application of, and impact of those principles remains a challenge for large transdisciplinary programs, particularly in agroecological systems where the social and environmental changes extend beyond a grant-funded project. To identify emerging project outcomes and significant changes through a principles-focused evaluation approach, our team is employing a mix of data collection strategies.
Hot Tips:
Principles can direct the design of focus groups and interview guides, provide a lens for data analysis, and inform key questions and group metrics of success in transdisciplinary groups. Ranging from developing perennial crops to conservation livestock grazing to integrating solar panels in farming, these agroecosystems projects bring a wide range of environmental disciplines, priorities, and stakeholder perspectives to the table. Here are four examples of data collection methods that we are adapting across these ongoing projects using principles-focused evaluation:
Evaluative Rubrics: Principles help illustrate the “best case scenarios”, articulating what research and partnerships that maintain project values would really look like. This allows evaluators to work backwards from that best case to develop evaluative rubrics, identifying different thresholds of effectiveness and success, and articulating key areas for growth.
Key Informant Interviews: We have asked project teams directly about their principles explicitly in key informant interviews. When we analyze those interviews, we look not only at their answers but also the implicit values that they share throughout the interviews, which gives us useful information about group shifts in perspective, understanding, and motivation in light of their project principles.
Most Significant Change: To illustrate the unintended or surprising effects of the projects, we’re using Most Significant Change to identify meaningful project events and activities through personal accounts and stories, particularly with stakeholder groups. Principles provide a key lens for data analysis to examine whether principles were adhered to in stories, and how principles shaped those experiences.
Ripple Effects Mapping: We are also using Ripple Effects Mapping to examine chains of causality between project activities, events, and partnerships, using principles to shape both the interview guide and data analysis. This group mapping process uses appreciative inquiry to highlight areas of strength, growth, and collaboratively examine how and where group principles were enacted over the course of the project, and where they affected project direction and leadership, stakeholder experiences, and research outcomes.
Thanks for reading! I hope these examples generate some ideas about data collection in principles-focused evaluation, using methods that involve program stakeholders and team members in identifying the significant project outcomes, effects, and stories.
The American Evaluation Association is hosting Environmental Program Evaluation TIG Week with our colleagues in the Environmental Program Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.