AEA365 | A Tip-a-Day by and for Evaluators

CAT | Advocacy and Policy Change

My name is Heather Krause.  As a data scientist for the Ontario Syrian Refugee Resettlement Secretariat, part of my job is to design ways to harness data to measure how successfully refugee resettlement is going, as well as what programs and services are working well and which ones have gaps.

Using data to advocate for vulnerable groups can be tricky.  For starters, not everyone in vulnerable groups is wild about the idea of having data collected on them.  Secondly, there is usually a broad range of stakeholders who would like to define success.  Thirdly, finding a comparison group can be challenging.

To avoid placing additional burden on vulnerable people, one option is to use public data such as Census, school board, or public health data.  This removes both the optical and practical problem of collecting data specifically from a unique or small population.  Public data can often be accessed at a fine enough level to allow for detailed analysis if you form partnerships and data sharing understandings with the public data owners.  An agreement to include their questions of interest in your analysis and to share your findings with these often-overburdened organizations goes a long way to facilitating data sharing agreements.

Once you have access to public data, deciding on indicators of success is the next step.  For example, accessing day care and working outside the home is seen as empowerment by some women, but not others.  Neither of these is a neutral measure of success.  To make matters more complex, diverse stakeholders often define success differently – from finding adequate housing to receiving enough income to not receiving social assistance.

Lesson Learned: I have found that the best way to handle this is to allow the voices of the vulnerable group to guide the foundation of how success is defined in the measurement framework.  Then to add a few additional indicators that align with key stakeholders’ interest.

Finally, once you have data and indicators selected you need to devise a way of benchmarking success with vulnerable groups.  If, for example, the income of refugees is being measured – how will we know if that income is high enough or changing fast enough?  Do we compare their income to the general population income?  To other immigrant income?  To the poorest community income?

Hot Tip: There is no simply answer.  The best way to deal with this is to build multivariate statistical models that include as many unique sociodemographic factors as possible.  This way you can test for differences both within and between many meaningful groups simultaneously.  This helps you avoid false comparisons and advocate more effectively for vulnerable populations using data.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Julia Coffman, and I am director of the Center for Evaluation Innovation, where we are building the field of evaluation in areas that are challenging to assess, including advocacy and policy change.

These are complex political times. Advocates are facing new challenges as they grapple with unpredictable developments that create political dysfunction and lessen the impact of once-effective tactics.

This uncertainty makes advocacy evaluation more important than ever. Advocates navigating uncharted waters need reliable feedback that helps them to learn and adjust as they go.

We advocacy evaluators need to be up to the task.

Hot Tip: Get ready to evaluate new strategies and tactics. 

Many advocacy evaluation efforts to date have focused on strategies (often legislative) using common tactics that assume a combination of persuasive research, public will-building, and bipartisan champion building will be enough to effect change.

Today the motivations of elected officials may have nothing to do with the rational selection of evidence-based policies that hold the most promise for constituents. Advocacy is changing to accommodate these new realities.

Rad Resources: The Atlas Learning Project offers resources on approaches that may be less common to advocates and their evaluators, but are expected to get more play in the current environment.

Hot Tip: Bone up on your political science.

Many of us studied political science in college, but have not kept up with it since. Political science is a discipline in which the answers regularly change to critical questions about how policy change happens or what motivates elected officials.

Learning the science behind what is happening in politics and why is critical as we pressure test advocacy theories of change and help advocates to select and measure outcomes that matter.

Rad Resources: Connect to the latest political science without going back to the classroom.

The Monkey Cage is a blog in The Washington Post that connects political scientists and their research to current events, helping to make sense of the “circus that is politics.”

Philanthropy in a Time of Polarization is an article in Stanford Social Innovation Review that explains why policy strategies used historically are no longer effective during this time of political polarization and hyper-partisanship.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Welcome to the Advocacy and Policy Change (APC) TIG week!  I’m Jewlya Lynn, the CEO at Spark Policy Institute. I am excited to kick-off the week in this year of political uncertainty and dynamic change. Our blog posts will explore timely, relevant insights regarding evaluation’s role in advocacy work around the world.

Change is inevitable whether your advocacy evaluation work is local, state, or federally focused or in the arts, education, justice, human services, equity, etc. Federal policies are changing the local and state funding available, as well as the constraints and opportunities for public and private institutions.

Advocates cannot ignore these changes.  Neither can evaluators.  But what is our role in this messy, dynamic environment?

HOT TIP: Kick into learning mode

“Accepting your limitations is every bit as important as embracing your strengths.” Dawn Jayne

Stay informed of what is going on in the political environment, learning with and from advocates. You may need to retool, acknowledging gaps in your skills as strategies shift. For example, your knowledge of evaluating inside game strategies may not translate fully to evaluating outside game strategies.

RAD RESOURCE: Point K Learning Center: You’ll find a wide range of top notch resources from leaders throughout the advocacy evaluation field.

HOT TIP: Help redefine success, but not too quickly

“The arrogance of success is to think that what you did yesterday will be sufficient for tomorrow.” William Pollard

Be prepared to help in the redefinition of what success looks like, even if figuring that out can’t happen quickly or easily. Help test underlying assumptions, engage in learning from experiments, and untangle the “why” behind small wins and losses. You can be a learning partner as advocates grapple with the new environment, helping to surface what wins might be possible, what success could look like.

HOT TIP: Resist the urge to be “right”

Advocacy partners are likely to have moments of confidence, moments of uncertainty, and a lot of moments in between. Be careful not to be too confident yourself – in your methods, the timing of when you want to deploy them, or even the accuracy of your findings. Flexibility isn’t just about being adaptive to the needs, it’s about acknowledging you don’t know how to best adapt and asking for help, from your advocacy partners and others.

RAD RESOURCE: APC TIG’s discussion board is a place to ask questions and seek new ideas

Major shifts in the political environment will happen and advocacy evaluators are lucky enough to be able to play an important learning role. But, it’s also just fine to put down the survey tool and pick up the protest sign, advocating for the change you want to see.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

We are Anne Gienapp and Sarah Stachowiak from ORS Impact, a consulting firm that helps organizations use data and evaluation to strengthen their impact, especially in hard-to-measure systems change efforts.

Ten years ago, when the field of advocacy and policy change was first coalescing, a number of excellent field building publications helped make the case for the value of theory of change, identification of interim outcomes, and the application of new tools and methods to fit the dynamic and adaptive space of advocacy efforts.

As the field has grown, so has the number of resources and frameworks that evaluators can use to deepen their evaluative practice in this space.  If you are like us, you probably have a “good intention” reading pile somewhere, where you have taken note of some of these as they were initially disseminated.  To round out the APC TIG week, we’ve listed three of our favorite newer resources that expand upon earlier work that helped define the field of advocacy evaluation.

Rad Resources

  • Beyond The Win: Pathways for Policy Implementation While early advocacy evaluation primarily focused on unique campaign wins, there has been increasing acknowledgement that understanding more than legislative wins would strengthen advocacy and policy change theories of change and evaluation designs.  The Atlas Project supported this publication to help identify ways in which to understand key aspects of policy implementation
  • Assessing and Evaluating Change in Advocacy Fields Early on, there was agreement that advocacy capacity could be a legitimate and important advocacy outcome.  Jewlya Lynn of Spark Institute   expands upon that notion with an evaluation framework for funders who recognize that a long-term strategy for meaningful and sustained policy change can include building the collective capacity and alignment of a field of individuals and organizations toward a shared broad vision.
  • Measuring Political Will: Lessons from Modifying the Policymaker Ratings Method While Julia Coffman and Ehren Reed’s original Unique Methods in Advocacy Evaluation first shared the idea of Policymaker Rating, there hasn’t been more public writing about it since.  This piece shares lessons learned about putting this method into practice in various circumstances and shares some things to do—and things to avoid—if you want to implement it.

This is certainly not an exhaustive list;  for more rad resources, be sure to check out the Center for Evaluation Innovation, Point K resource page and the Atlas Project website.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Carlisle Levine, President and CEO of BLE Solutions, LLC. We offer evaluation, applied research and technology services to help organizations increase their effectiveness and contribute to better outcomes. I specialize in global advocacy, peacebuilding and strategic evaluation.

A tremendous challenge in advocacy evaluation is identifying links between advocacy activities and changes in people’s lives, given the many factors that are involved and the time it takes for change to come about. The Most Significant Change approach can help respond to this challenge.

Rad Resource: The Most Significant Change (MSC) approach, an inductive, participatory outcome monitoring and evaluation approach, was developed by Rick Davies and then widely publicized in a guide co-authored with Jess Dart. It uses storytelling to gather evidence of intended and unintended, as well as positive and negative change. The stories are then reviewed and analyzed by a core team to identify the most significant change from their point of view. Importantly, MSC is not a standalone method. Rather, it can point to outcomes that require further validation using more deductive methods.

The approach involves 10 steps, according to the MSC Guide:

MSCStepsGraphic.Levine

Lessons Learned

  • In evaluating advocacy efforts, I first use methods that help me identify the contribution that advocacy efforts have made to policy changes. I then use MSC to explore early evidence of how those policy changes are affecting people’s lives.
  • In my design, I do not define domains of change, but wait to see what domains emerge from the stories themselves.
  • By triangulating a storyteller’s story with information provided by people familiar with the storyteller’s life, I increase the story’s credibility.
  • With my clients, I use the selection process to help them understand the variety of changes in people’s lives resulting, at least in part, from their targeted policy change. I also conduct a meta-analysis that shows them trends in those changes. With this information in hand, they can reinforce or adjust their policy goals and advocacy efforts in order to contribute to the types of change they most desire.

Hot Tip: To build trust with storytellers, I partner with story collectors who speak their language and are familiar with their context. The more storytellers believe a story collector can relate to their reality and will not judge them for it, the more open storytellers will be.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, I’m Oscar Espinosa from Community Science . We recently evaluated the effectiveness of professional development programs in various sectors that seek to diversify their leadership or workforce to be more responsive to communities of color.

Hot Tips

  • Specify what program effectiveness means–to all stakekholders! A program’s intended objectives are oftentimes skewed to the perspective of the funder. As an evaluator, you need to consider the various program stakeholders and determine what effectiveness looks like for each of them. To that end, sessions to develop program logic models should be held with the funder and separately with other program stakeholders. Vetting and reconciling the models is an essential step to establish a good foundation, before moving on to an evaluation design. Allocate enough time for this process as reaching consensus can be a laborious task.
  • Capture participants’ accomplishments but don’t downplay challenges. Despite pressures from funders, who understandably want to highlight positive impacts, as an evaluator you have to identify unintended program consequences and areas for improvement. Data collection needs to focus on challenges participants experienced, including perceptions that activities were not tailored to people of color or their cultural or linguistic needs. Be prepared to have uncomfortable discussions about structural racism or equity issues. Doing this can lead to solid recommendations for program improvement.
  • Numbers and stories are BOTH essential. We were interested in what brought participants to the program; their expectations as compared to their actual experience; and the influence the program had on them. We found that a combination of forced-response survey items and open-ended, semi-structured interviews before and after participants complete the program were effective methods for getting a full picture.

Lesson Learned: To effectively evaluate professional development programs, one needs to take into account both funding organizations’ policies and culture and people’s of color needs and background.  The evaluators’ art is their ability to extract the voice of program participants from the noise produced by program requirements and the institutional context. Ultimately, a program’s effectiveness should be judged on the extent to which is motivates people of color to continue to take on new challenges and advance in their profession.

Rad Resources

  • Handbook on Leadership Development Evaluation is a comprehensive resource filled with examples, tools, and the most innovative approaches to evaluate leadership development in a variety of settings.
  • L. Kirkpatrick’s Evaluating Training Programs focuses on evaluation approaches to measuring reaction, learning, behavior, and results.
  • Special Issue: Building a New Generation of Culturally Responsive Evaluators through AEA’s Graduate Education Diversity Internship Program.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, we are Kat Athanasiades and Johanna Morariu from Innovation Network, an evaluation firm that works with philanthropic and nonprofit organizations, especially those engaged in advocacy.

Advocacy evaluation has been a difficult field to generate shareable lessons. Many organizations and campaigns are concerned that sharing their “secrets” (information usually divulged in evaluation reports such as methods, approaches, and incremental wins) will give the “enemy” valuable information that might undermine future phases of their work. Given this dearth, we saw an opportunity to try to support more general learning. We sought to highlight this process by looking at the past ten years of The Atlantic Philanthropies’ immigration reform advocacy grantmaking in the US. We used this campaign as a case study to highlight important decision points that a broader audience of nonprofits, funders, and evaluators could apply to their own work, regardless of the issue.

Hot Tip: Get more mileage out of your evaluations! With good planning, one evaluation may result in a paper for the board, a blog post for the evaluation community, and a visualization to send out over Twitter. It’s not always feasible to do this, given budget and confidentiality constraints, but when you can, do it! You will add to the advocacy evaluation field and contribute to improved practice among advocates.

Our evaluation project resulted in a fairly traditional report, documenting the history of US federal immigration reform. The report is most likely to be used by close-in project stakeholders (Atlantic and immigration reform advocates), but to expand the relevance of strategic lessons to a broader audience, we pulled Atlantic’s key decisions out of this report and then elaborated the implications, pros, and cons, around each decision. We also developed discussion questions that evaluators can use as a facilitation guide with partners who are considering advocacy work.

Rad Resource: We’ve also created a Funder Discussion Guide to accompany the traditional report. As an example, one decision that any advocacy funder may make is whether to emphasize grassroots or grasstops funding strategies. The decision partly rests on whether the issue of interest has become politicized and what opportunities current political realities afford. Using the questions in the Guide, an evaluator can help walk an advocacy partner through a theory of change process, thinking through the context, assumptions, and needs that underlie their work.

fdg_cover_image

Lesson Learned: Share lessons! We started this post by explaining that there’s a dearth of information in the advocacy evaluation field. We invite you all to share how you have made use of evaluation projects to expand learning and use of your findings.

Image via

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We are Rhonda Schlangen (US), Julie Tumbo (Tanzania), and Ben Awinda (Tanzania), evaluation consultants specializing in advocacy and development.  In Tanzania and other countries, civil society advocates struggle with developing skills and finding support for their efforts. We’re using the dissemination phase of an evaluation of three groundbreaking Tanzanian advocacy campaigns to support advocate capacity-building. This encouraged us to rethink the usual presentation-and-distribution-of-reports approach and conceptualize dissemination as a platform on which to build learning.

Lesson learned: Take a campaign approach to dissemination. For this project, we designed the Mwanaharakati (“Activist”) Campaign to both share the evaluation results and involve civil society actors, particularly those in Tanzania, in conversations about the case studies and use of the information to advance their own advocacy work.

Hot tips:

  • Start designing the dissemination campaign plan by first asking advocates how they and their constituencies would like to receive the information. We discovered that Facebook is out, WhatsApp is in, and everyone loves cartoons.
  • Keep the conversation going! Ben poses biweekly questions related to issues highlighted in the case studies on the campaign’s social media accounts in both Kiswahili and English.

Rad Resource: Graphic novels!  We wanted to reach young people, individuals with limited literacy abilities, and people with limited patience to read long documents, so we worked with a talented cartoonist to develop comic versions of the case studies. It was important to work with a local artist who understood the political context and the cultural nuances of images and colors. He accurately and compellingly conveyed the advocacy issues and tactics.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Welcome to the Advocacy and Policy Change (APC) TIG week on AEA365!  I’m Jared Raynor, Director of Evaluation at TCC Group and co-chair of the TIG.  Our TIG is celebrating our 10-year anniversary at this fall’s AEA conference.  This week’s blog posts share some of the great insights gained regarding evaluation’s role in advocacy work around the world.

In preparation for the TIG week, I asked for some reflections from some people who were with me when the TIG formed.  Tom Kelly’s had a striking insight: “We’d always said we are not inventing any new tools of evaluation but were looking for ways that evaluation can be applied in complex, rapidly changing policy advocacy environments—although look at the new tools that have come along.”  To start the APC week, I wanted to reflect on a few of the amazing developments in our field.

Rad Resource: Over the course of the TIG’s development we’ve been asked on occasion how to share resources within the AEA community.  On each occasion, we have opted to promote existing aggregators of information.  Innovation Network’s Point K Learning Center has consistently gathered resources from the field and The Center for Evaluation Innovation (CEI) has supported the development of new material. Both make the information freely available.

Rad Resource: One of the early pieces of writing on advocacy evaluation, The Challenge of Assessing Policy and Advocacy Activities, remains a great starting place.  The authors identify seven key challenges faced by foundations in advocacy evaluation, including complexity, role of external forces, timeframe, shifting strategies and milestones and attribution.  More recently, the Overseas Development Institute did a comprehensive review of Monitoring and Evaluation of Policy Influence and Advocacy that looked at trends, approaches, frameworks and methods for evaluating advocacy.  And, coming out later this year is the first book on advocacy evaluation by Annette Gardner!

Lessons Learned: In late 2015, the Aspen Planning and Evaluation Program and CEI convened a small group of advocacy evaluators to review the state of the field. I want to share three things that struck me from that conversation.  First, advocacy evaluators need to become more savvy at eliciting theories of change alongside theories of action.  We are fairly adept at the latter and frequently let the former slide as too abstract.  Second, we should continue to push ourselves to incorporate counterfactual thinking into evaluations.  Third, we should constantly consider the political implications of our work—how it is positioned, whose voice is prioritized, and what bias we bring to the advocacy work.

We have come a long way and I look forward to where we as a field go next!

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi! I’m Lisa Hilt, a Monitoring, Evaluation, and Learning Advisor for Policy and Campaigns at Oxfam.

We strive for policy changes that will right the wrongs of poverty, hunger, and injustice. Much of our progress takes place in small steps forward, resulting from ongoing engagement with key stakeholders and multiple campaign spikes (high intensity, short-term advocacy moments focused on a particular issue).

Following these campaign spikes, teams ask:

  • Were the outcomes worth the resources we invested?
  • How can we be more effective and efficient?

We evaluators ask: How can we support teams to answer these questions with confidence when in-depth analyses are not possible or appropriate? We’ve found from our experience at Oxfam that conducting “simple” value for money analyses for campaign spikes is a useful alternative for the teams we support.

Here are a few tips and lessons based on our experience:

Hot Tips:

Plan ahead: Even simple analysis can be difficult (or impossible) to conduct without pre-planning. Decide in the planning phases of the campaign spike which indicators and investments will be tracked and how.

Break down investments by tactic: Having even a high level breakdown of spending and staff time by key tactics (see example) enables more nuanced analysis of the connections between particular investments and the intended outcomes.

Team analysis is key: In addition to using “hard” data as a source of evidence, utilize insights of team members who bring multiple perspectives and are experts in their field to assess the extent to which their interrelated efforts relate to the results. Team debriefs are an effective way to do this.

Hilt

Lessons Learned:

Present information visually: A visual presentation of investments and outcomes enhances the team’s ability to make sense of the information and generate actionable insights (see example). Indicate which tactics were intended to achieve specific objectives.

Don’t let perfection be the enemy of the good: Slightly imperfect analysis is better than no analysis at all, and often adequate for short-term campaign spikes. Match the levels of rigor and effort to the confidence level needed to enable the team to generate reliable insights.

Trust is important: Trust and communication is fundamental to honest conversations within the team. Be cognizant of team dynamics when designing team reviews, and focus the discussion on the outcomes and tactics, not individual performance.

Focus on the future: The strategic learning and forward-looking aspects of this type of exercise are arguably the most important. While looking back at the campaign spike, focus the conversation on what the team can learn from this experience to improve future efforts.

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

<< Latest posts

Older posts >>

Archives

To top