Hi, I’m Giovanni Dazzo, evaluator with the Department of State’s Bureau of Democracy, Human Rights, and Labor (DRL). As an evaluator embedded within a program office, I often hear questions from our staff and grantees on whether we are valuing the right outcomes or measuring what matters.
In this post, I’ll outline how we questioned the way we were interpreting outcomes by dissecting power and values within the organizational structures we were working within and the methods we were using.
Lessons Learned: Over the last two years, we have been using Most Significant Change (MSC)—a qualitative, participatory approach for collecting stories with project participants and grantees—to understand intended and unintended changes, both positive and negative. As we started reviewing and selecting stories during our selection panels at DRL, we realized it would be important to question which outcomes we were valuing most and how we were determining significance. This is especially helpful as questions of power arise when donors organize panels to select the outcomes that are “most significant”.
Hot Tip:
We applied a critical frame to two steps within the MSC methodology—secondary analysis and meta-monitoring—to explore power dynamics and values in our process. This included qualitatively coding all stories to understand whose stories were being told and the types of outcomes mentioned. We also analyzed panelists’ remarks during the final story selection panel to explore which outcomes we, as the donor, determined as significant. Through interviews with grantees, we were also able to learn if they valued stories or outcomes differently than we did.
For instance, in the democracy and rights space, we often value big wins like policy reform. However, in selecting these outcomes as the most significant, we realized that we may be losing sight of the changes felt most by our project participants—such as building community acceptance, being able to express oneself, or forming movements and alliances.
Lessons Learned:
- Question why you are measuring or evaluating certain constructs and ask why or to whom they are significant.
- Question your methods. Simply because a particular method may be framed as participatory does not mean you are applying it in a way that values diverse voices, histories, and social structures.
- From an evaluative standpoint, using MSC and coding stories allowed us to map intermediary outcomes within democracy and human rights projects, especially those that are often overshadowed by big wins.
Rad Resources:
- The Most Significant Change Guide, co-authored by Rick Davies and Jess Dart, provides an accessible overview of the steps needed to employ this methodology and how you can make it work for your organization.
- If you’re interested in reviewing our process, check out AEA’s Public eLibrary and search for ‘Whose Stories Matter?’.
The American Evaluation Association is celebrating Democracy & Governance TIG Week with our colleagues in the Democracy & Governance Topical Interest Group. The contributions all this week to aea365 come from our DG TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by theAmerican Evaluation Association and provides a Tip-a-Day by and for evaluators.