APC TIG Week: Carlisle Levine on Using the Most Significant Change Approach to Explore Potential Evidence of Impact in Advocacy and Policy Change

Hello! I’m Carlisle Levine, President and CEO of BLE Solutions, LLC. We offer evaluation, applied research and technology services to help organizations increase their effectiveness and contribute to better outcomes. I specialize in global advocacy, peacebuilding and strategic evaluation.

A tremendous challenge in advocacy evaluation is identifying links between advocacy activities and changes in people’s lives, given the many factors that are involved and the time it takes for change to come about. The Most Significant Change approach can help respond to this challenge.

Rad Resource: The Most Significant Change (MSC) approach, an inductive, participatory outcome monitoring and evaluation approach, was developed by Rick Davies and then widely publicized in a guide co-authored with Jess Dart. It uses storytelling to gather evidence of intended and unintended, as well as positive and negative change. The stories are then reviewed and analyzed by a core team to identify the most significant change from their point of view. Importantly, MSC is not a standalone method. Rather, it can point to outcomes that require further validation using more deductive methods.

The approach involves 10 steps, according to the MSC Guide:

MSCStepsGraphic.Levine

Lessons Learned

  • In evaluating advocacy efforts, I first use methods that help me identify the contribution that advocacy efforts have made to policy changes. I then use MSC to explore early evidence of how those policy changes are affecting people’s lives.
  • In my design, I do not define domains of change, but wait to see what domains emerge from the stories themselves.
  • By triangulating a storyteller’s story with information provided by people familiar with the storyteller’s life, I increase the story’s credibility.
  • With my clients, I use the selection process to help them understand the variety of changes in people’s lives resulting, at least in part, from their targeted policy change. I also conduct a meta-analysis that shows them trends in those changes. With this information in hand, they can reinforce or adjust their policy goals and advocacy efforts in order to contribute to the types of change they most desire.

Hot Tip: To build trust with storytellers, I partner with story collectors who speak their language and are familiar with their context. The more storytellers believe a story collector can relate to their reality and will not judge them for it, the more open storytellers will be.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

3 thoughts on “APC TIG Week: Carlisle Levine on Using the Most Significant Change Approach to Explore Potential Evidence of Impact in Advocacy and Policy Change”

  1. Kirsten Fritsch

    Carlisle,

    Thank you for your post. As part of my degree work through Queen’s University, Ontario I am to respond to an article of personal interest. I am currently wrapping up an evaluation on a social program in my area called Best Beginnings Baby and Me (BBBM). Our task, in our course Program Evaluation, was to find a program that we considered meaningful and use the framework and research we uncover throughout the course to create a Program Evaluation. As such, I chose BBBM because I feel strongly that this program is entirely beneficial. I also feel that it leaves me in greater position to connect with and collect the stories of participants as I use the Most Significant Change model.

    Your hot tip about building trust with the storytellers resonates with me as I was part of this program with the mothers that I interviewed, while I was on maternity leave. These mothers understand that I am able to relate to them and do not judge them because I have also been a participant in BBBM within the last year. Related, though offtopic, is the discussions that I have had with these mothers about how it seems as though only other mothers can truly understand our everyday struggles. Also, as you point out, I can speak their language, or the vernacular of mothers, if you will. Discussions about issues related to their lives and this program will probably include language that an evaluator from outside of this context may not understand.

    Most helpful in your article is the framework you provide of the ten steps of MSC approach. Unfortunately, I am unsure how to verify the stories of these mothers. I know that they are credible sources because I have a personal relationship with the majority of mothers I interviews. However, without verification are these stories still impactful and useful in an evaluation?

    I have also considered that my stories of MSC may not create action or revise the system as the program being evaluated has been in place in a small town for many years. If there is something that does not happen easily in a small town, it is change, regardless of the evidence. However, at this point of the evaluation the stories have proved the impact of the program in improving the lives of expectant and new mothers.

    As you discuss, it is an effective way to measure advocacy work. In this case, the work being done is “to improve the lives of expectant and new mothers living in conditions of risk.” Improvement is difficult to measure without testimony of participants.

    I also appreciate that this approach is not enough on its own. There are other data collection and evaluation methods that can enhance and support the stories of MSC.

    Thank you for your article and I appreciate the opportunity to respond!

  2. Kirsten Fritsch

    Carlisle,
    Thank you for your post. As part of my degree work through Queen’s University, Ontario I am to respond to an article of personal interest. I am currently wrapping up an evaluation on a social program in my area called Best Beginnings Baby and Me (BBBM). Our task, in our course Program Evaluation, was to find a program that we considered meaningful and use the framework and research we uncover throughout the course to create a Program Evaluation. As such, I chose BBBM because I feel strongly that this program is entirely beneficial. I also feel that it leaves me in greater position to connect with and collect the stories of participants as I use the Most Significant Change model.
    Your hot tip about building trust with the storytellers resonates with me as I was part of this program with the mothers that I interviewed, while I was on maternity leave. These mothers understand that I am able to relate to them and do not judge them because I have also been a participant in BBBM within the last year. Related, though offtopic, is the discussions that I have had with these mothers about how it seems as though only other mothers can truly understand our everyday struggles. Also, as you point out, I can speak their language, or the vernacular of mothers, if you will. Discussions about issues related to their lives and this program will probably include language that an evaluator from outside of this context may not understand.
    Most helpful in your article is the framework you provide of the ten steps of MSC approach. Unfortunately, I am unsure how to verify the stories of these mothers. I know that they are credible sources because I have a personal relationship with the majority of mothers I interviews. However, without verification are these stories still impactful and useful in an evaluation?
    I have also considered that my stories of MSC may not create action or revise the system as the program being evaluated has been in place in a small town for many years. If there is something that does not happen easily in a small town, it is change, regardless of the evidence. However, at this point of the evaluation the stories have proved the impact of the program in improving the lives of expectant and new mothers.
    As you discuss, it is an effective way to measure advocacy work. In this case, the work being done is “to improve the lives of expectant and new mothers living in conditions of risk.” Improvement is difficult to measure without testimony of participants.
    I also appreciate that this approach is not enough on its own. There are other data collection and evaluation methods that can enhance and support the stories of MSC.
    Thank you for your article and I appreciate the opportunity to respond!

Leave a Reply to Kirsten Fritsch Cancel Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.