AEA365 | A Tip-a-Day by and for Evaluators

CAT | Advocacy and Policy Change

Welcome to the Evaluation 2013 Conference Local Arrangements Working Group (LAWG) week on aea365. I’m Will Fenn from Innovation Network a Washington, D.C.-based monitoring and evaluation consulting firm specializing in advocacy and policy change evaluation.

There is an all too common situation that arises around evaluation — I’ll call it the evaluation merry-go-round. I saw this situation many times as a foundation program officer. Now that I work in a role fully focused on evaluation, my goal for the “State of Evaluation Practice” is to help organizations avoid the merry-go-round and promote evaluation that embraces data-based decisions and learning.

Lesson Learned—Let me explain how the evaluation merry-go-round often starts: A funder recognizes the importance of evaluation at a board meeting and approaches its grantees requesting data for an evaluation in the coming year. If the grantee has good data, evaluation moves along happily for both parties. But often resources are tight for grantees and they were not able to capture good quality data even if they are doing great work. The grantee offers what they have, the funder may then question the data, accept the incomplete data, or perform their own data collection with the evaluation conclusion sent to the board.

The process is often uncomfortable for both sides and too often leaves grantees in no better position to improve operations through data informed decision-making. In other words, funder and grantee go up and down through the evaluation process but the ride ends with the organization in the same place.

Hot Tip: My experience is that the same scenario can play-out successfully when funders and grantees cooperate, plan, and invest from the earliest stage to build capacity before the evaluation. A high level of engagement and planning from both parties is essential; and additional resources in terms of funding and expertise are highly recommended. Remember, data is not king; it only helps one ask the right questions. There is no substitute for investing time to understand the context around the data and to know which data to consider.

Rad Resources:

The Stanford Social Innovation Review article, Counting Aloud Together, shows an example of how to build evaluation capacity.

The Collaborative Evaluation” section in the Learning from Silicon Valleyarticlealso offers great tips from the Omidyar Network’s experience on collaborative evaluation.

Also check out Innovation Network’s guide to evaluation capacity building.

saloonHot Tip—Insider’s advice for Evaluation 2013 in DC: For a quiet place to reflect on the day’s events visit the Saloon at 1205 U Street, NW. The bar’s mission is to promote conversation, so it is free of TV screens and offers large communal tables upstairs.

We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). AEA is accepting proposals to present at Evaluation 2013 through until March 15 via the conference website. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

No tags

We are Colleen Duggan, Senior Evaluation Specialist, International Development Research Centre (Canada) and Kenneth Bush, Director of Research, International Conflict Research (Northern Ireland).  For the past three years, we have been collaborating on a joint exploratory research project called Evaluation in Extremis:  The Politics and Impact of Research in Violently Divided Societies, bringing together researchers, evaluators, advocates and evaluation commissioners from the global North and South. We looked at the most vexing challenges and promising avenues for improving evaluation practice in conflict-affected environments.

CHALLENGES Capture1Conflict Context Affects Evaluation – and vice versa.  Evaluation actors working in settings affected by militarized or non-militarized violence suffer from the typical challenges confronting development evaluation.  But, conflict context shapes how, where and when evaluations can be undertaken – imposing methodological, political, logistical, and ethical challenges. Equally, evaluation (its conduct, findings, and utilization) may affect the conflict context – directly, indirectly, positively or negatively.

Capture

Lessons Learned:

Extreme conditions amplify the risks to evaluation actors.  Contextual volatility and political hyper-sensitivity must be explicitly integrated into the planning, design, conduct, dissemination, and utilization of evaluation.

  1. Some challenges may be anticipated and prepared for, others may not. By recognizing the most likely dangers/opportunities at each stage in the evaluation process we are better prepared to circumvent “avoidable risks or harm” and to prepare for unavoidable negative contingencies.
  2. Deal with politico-ethics dilemmas. Being able to recognize when ethics dilemmas (questions of good, bad, right and wrong) collide with political dilemmas (questions of power and control) is an important analytical skill for both evaluators and their clients.  Speaking openly about how politics and ethics – and not only methodological and technical considerations – influence all facets of evaluation in these settings reinforces local social capital and improves evaluation transparency.
  3. The space for advocacy and policymaking can open or close quickly, requiring readiness to use findings posthaste. Evaluators need to be nimble, responsive, and innovative in their evaluation use strategies.

Rad Resources:

  • 2013 INCORE Summer School Course on Evaluation in Conflict Prone Settings , University of Ulster, Derry/ Londonderry (Northern Ireland. A 5-day skills building course for early to mid-level professionals facing evaluation challenges in conflict prone settings or involved in commissioning, managing, or conducting evaluations in a programming or policy-making capacity.
  • Kenneth Bush and Colleen Duggan ((2013) Evaluation in Extremis: the Politics and Impact of Research in Violently Divided Societies (SAGE: Delhi, forthcoming)

The American Evaluation Association is celebrating Advocacy and Policy Change (APC) TIG Week with our colleagues in the APC Topical Interest Group. The contributions all this week to aea365 come from our APC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hello – we’re Claire Hutchings and Kimberly Bowman, working with Oxfam Great Britain (GB) on Monitoring, Evaluation and Learning of Advocacy and Campaigns. We’re writing today to share with you Oxfam GB’s efforts to adopt a rigorous approach to advocacy impact evaluation and to ask you to help us strengthen our approach.

Rad Resources Resources:

As part of Oxfam GB’s new Global Performance Framework, each year we randomly select and evaluate a sample of mature projects.  Project evaluations that don’t lend themselves to statistical approaches, such as policy-change projects,   are particularly challenging. Here, we have developed an evaluation protocol based on a qualitative research methodology known as process-tracing.  The protocol attempts to get at the question of effectiveness in two ways: by seeking evidence that can link the intervention in question to any observed outcome-level change; and also by seeking evidence for alternative “causal stories” of change in order to understand the significance of any contributions the intervention made to the desired change(s).  Recognizing the risks of oversimplification and/ or distortion, we are also experimenting with the use a of simple (1-5) scale to summarize the findings.

Lessons Learned (and continuing challenges!):

  • As a theory based evaluation methodology, process tracing involves understanding the Theory of Change underpinning the project/campaign, but this is rarely explicit – and can take time to pull out.
  • It’s difficult (and important) to Identify ‘the right’ interim outcomes to focus on.  They shouldn’t be very close in time and type to the intervention; that could make the evaluation superfluous.  Nor should the outcomes be so far down the theory of change that they can‘t realistically occur or be linked causally to the intervention within the evaluation period.
  • In the absence of a “signature” – something that unequivocally supports one hypothesized cause – what constitutes credible evidence of the intervention’s contribution to policy change?  Can we overcome the charge of (positive) bias so often leveled at qualitative research?

And of course, all this coupled with the very practical implementation challenges!  The bottom line: like all credible impact evaluations, it takes time, resources, and expertise to do these well. We have to balance real resource and time constraints with our desire for quality and rigor.

As we near the end of our second year working with this protocol, we are looking to review, refine, and strengthen our approach to advocacy evaluation.  We would welcome your inputs! Please use the comments function below or blog about the issue to share your experience and insights, “top tips” or “rad resources.”  Or email us directly.

The American Evaluation Association is celebrating Advocacy and Policy Change (APC) TIG Week with our colleagues in the APC Topical Interest Group. The contributions all this week to aea365 come from our APC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Rhonda Schlangen, and I’m an independent evaluation consultant specializing in advocacy and campaign evaluation.

Non-governmental service delivery organizations use advocacy as a lever for greater impact.  They often have a tough time finding useful internal evaluation systems to assess their combined service delivery and advocacy work.  Many service providers try to fall back on established evaluation processes for service delivery M&E—logic models, quantitative client counts—with disappointing results.  Advocacy evaluation is a dynamic and innovative corner of the evaluation world, and offers strategies that can improve M&E of both service delivery and advocacy.

Lesson Learned:

Strategies that work well with advocacy monitoring and evaluation position evaluation as a driver of effectiveness, capitalizing on the critical thinking skills of advocates and evaluators.  Repositioning evaluation as a tool for knowledge generation is particularly critical for advocacy.  Making evaluation more accessible and relevant to those doing the work will likely benefit service delivery as well.

Hot Tips:

  • PLANNING:  Link planning for advocacy and service delivery, but plan for different timeframes of change.  Advocacy will likely be cyclical and ongoing, with service delivery on a more linear path.
  • IMPLEMENTING:  Monitoring implementation with reflective processes focusing on progress, rather than outputs, provides a rich opportunity for joint advocacy/services review that is mutually informing.  Advocacy M&E requires consistent and regular review of, and response to, information.  Service delivery M&E could benefit from those processes as well.
  • EVALUATING:  Add innovative methods to the evaluation mix.  Newer approaches like Outcome Mapping or Outcome Harvesting place social change as a broad outcome to which services and advocacy are contributing strategies.

Rad Resources:

The American Evaluation Association is celebrating Advocacy and Policy Change (APC) TIG Week with our colleagues in the APC Topical Interest Group. The contributions all this week to aea365 come from our APC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Dr. Edmina Bradshaw, Principal Consultant of Edmina Bradshaw LLC.  I work with a small group of professionals on program planning and evaluation for national and local nonprofit organizations, mostly in the aging field.  I also support organizations in implementing self-directed volunteer teams(sdvnetwork.com) across the country.

I find that while the need for evaluation has finally gained traction with non-profits in general, evaluation planning is lagging way behind. In the past three years I have been called in to evaluate advocacy efforts in particular only to find there is no advocacy plan in place, outcomes are defined badly (or not at all) and “evidence” presented is at best anecdotal.  What can be done to help in such situations?

Hot Tips:

Help the organization retrace its steps through a summative evaluation process – to arrive at a collection of measurements and judgments that would allow conclusions to be drawn about outcomes and impacts of their advocacy program.

Lessons Learned:

  • Use a basic logic model to articulate the theory of change. Helps clarify short, medium and longest-term goals …. and how to get there.
  • Keep it simple – To begin the process, I use a checklist of the logic model elements: Inputs, Output (Activities, Participation), Outcomes (Short, Medium & Long-term), Assumptions and External Factors.  Organizations not used to theory of change find it less intimidating to tackle these separately at first – especially as different people may be involved in different stages of their program.
  • If the organization decides to continue its advocacy efforts – as is usually the case – use the results of the summative process as part of the baseline for a new formative evaluation process.

Bonus

  • After they see the final product of a succinctly articulated theory of change, our clients are thrilled and use it in reports, funding requests, etc. It is also a huge motivator to plan more effectively for the next iteration of their advocacy efforts.

Rad Resources:

  • Marcia Egbert and Susan Hoechstetter: “Evaluating Nonprofit Advocacy Simply: An Oxymoron?”  Harvard Family Research Project: Volume XIII, Number 1&2, Spring 2007

The American Evaluation Association is celebrating Advocacy and Policy Change (APC) TIG Week with our colleagues in the APC Topical Interest Group. The contributions all this week to aea365 come from our APC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello! I’m Carlisle Levine, an independent evaluator specializing in organizational and advocacy evaluation. I have led CARE USA’s advocacy evaluation and co-led Catholic Relief Services’ program evaluation.

As an internal and independent evaluator, I have learned important lessons about advocacy evaluation.

Lessons Learned:

  • Influencing policy change is a lengthy, convoluted process involving many actors, actions and events. Isolating the influence of any one element on a policy change is difficult. Thus, most advocacy evaluators look for evidence of contribution, not attribution.
  • Since policy change takes time, identifying meaningful measures of progress is important. Even in difficult political environments, advocates may be organizing and honing messages. Thus, incremental objectives should include measures related to advocacy capacity, as well as policy change.
  • Because advocates are very busy and often skeptical about monitoring and evaluation, monitoring activities must have immediate relevance to their work and require minimal effort on their part.
  • Advocates are often operating with limited funding. Identifying inexpensive monitoring and evaluation methods is critical.

Hot Tips and Rad Resources:

  • The Advocacy and Policy Change Composite Logic Model, developed by the Harvard Family Research Project, provides excellent guidance on meaningful, incremental measures of progress.
  • One possible desired outcome is a change in policymaker support for an issue.
    • The Policymaker Rating Tool, also developed by the Harvard Family Research Project and found in Unique Methods in Advocacy Evaluation by Julia Coffman and Ehren Reed, is a useful and simple tool to assess policymaker’s influence over a policy and his/her level of support for an issue.
    • At CARE, we worked with the Aspen Institute’s Advocacy Planning and Evaluation Program to create a policymaker champion scorecard that built on the Policymaker Rating Tool and identified specific measures relevant to CARE’s advocacy.  By April, the tool will be available as part of BOND’s Improve It Framework.  You can also read about the tool here.
    • Media is often part of an advocacy effort and media monitoring is useful for assessing its effectiveness. However, advocates may lack access to media monitoring tools. At CARE, our media team developed a media champions scorecard that relied on staff knowledge to assess outlets based on their support for CARE and its advocacy issues. This tool will also be available in April as part of BOND’s Improve It Framework.
    • Many organizations use periodic reviews, after-action reviews or Intense Period Debriefs to undertake internal assessments of advocacy progress. During these, advocates reflect on an advocacy initiative’s progress. In the conversation, an evaluator can help ask the right monitoring questions, and a trusted outsider can ground truth from the advocates’ claims and provide a different perspective.

The American Evaluation Association is celebrating Advocacy and Policy Change (APC) TIG Week with our colleagues in the APC Topical Interest Group. The contributions all this week to aea365 come from our APC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Welcome to Advocacy and Policy Change TIG week on AEA365!  I’m David Devlin-Foltz and I co-chair our TIG when I’m not directing the Advocacy Planning and Evaluation Program at The Aspen Institute or, well, doing other things.  My team and I work principally with foundations and nonprofits to assess efforts to advocate for policy change or behavior change.  We know that some AEA365 readers may share the view I heard at my first AEA meeting: “Advocacy?  You can’t evaluate that!”  As you’ll see in this week’s posts, APC TIG members disagree – mostly.  That is, you’ll hear about principles, tools and approaches that help us reduce our clients’ uncertainty about what works and what doesn’t.  And you’ll hear cautionary tales about the residue of uncertainty that will always remain.

Hot Tips: What to watch for this week

Tuesday:  Carlisle Levine will acknowledge some core advocacy evaluation challenges and provides some solutions.

Wednesday: Edmina Bradshaw offers a familiar evaluators’ lament and pragmatic responses.

Thursday: Rhonda Schlangen reminds us that advocacy and service delivery are related and mutually supportive; evaluation can be, too.

Friday: Oxfam GB’s Claire Hutchings and Kimberley Bowman brief us on how Process Tracing helps Oxfam answer some tough questions; and they invite aea365 readers to apply their collective brainpower in response.

Saturday: Colleen Duggan and Kenneth Bush from IDRC discuss the special challenges of evaluating change processes in settings of violent conflict.

Hey – we never said it was easy to assess the impact of efforts to encourage complex policy change and social change, and these last two posts underscore that.  But it’s possible; advocates, their funders and other stakeholders naturally want to know what works.  We’re on it.

Rad Resources:

  • This recent blogpost from Steven Mayer at JustPhilanthropy defines advocacy and encourages foundations to pay more attention to funding and assessing advocacy efforts.
  • Innovation Network shares its own tools and a host of resources at Point K, its web-based hub
  • Check out past AEA coffee break webinars on advocacy topics and listen in to Sue Hoechstetter’s webinar this Thursday, 2:00 – 2:20 PM EST, on her new advocacy capacity measurement tool: www.BolderAdvocacy.org

The American Evaluation Association is celebrating Advocacy and Policy Change (APC) TIG Week with our colleagues in the APC Topical Interest Group. The contributions all this week to aea365 come from our APC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hey there!  I’m Robert Medina, Program Manager for the Aspen Institute’s Advocacy Planning and Evaluation Program, a consulting group based in Washington, DC.

Rad Resource – So What? Your Weekly Guide to Advocacy with Impact: Most evaluators would agree that all data isn’t necessarily useful data.  To understand the distinction—between what’s valuable and what’s not, what should be measured and what doesn’t need to be—we like to ask that pesky question, “so what?”  It’s in that spirit that I blog each Friday.

Every week I write about research, news and cool resources related to advocacy and social change evaluation for an audience of advocates, evaluators, funders and others in the civil society sector.  Curious?  Sign up to receive new posts (just three short items) fresh off my keyboard.

 

Hot Tips – favorite posts: Here are five of my favorites.

  • Building capacity: We frequently tell advocates and funders that building advocacy capacity is just as important as hitting your policy targets.  Soon after the Komen debacle, Planned Parenthood showed exactly why this is the case.
  • Advocacy networks: In June 2012, Professor Jeremy Shiffman stopped by the Institute to talk about his work on global health policy networks.  I discuss how he evaluates the potential effectiveness of these advocacy networks based on issue characteristics, the socio-political environment and network-specific factors.
  • Telenovela advocacy: Ok, I admit it—I love telenovelas.  The more melodramatic, the better.  Fortunately, millions around the world agree with me.  So why not use this hugely popular medium to promote social issues, like education and HIV/AIDS prevention?  The Population Media Center and others conducting media-driven advocacy know it can work.
  • Social network analysis: In its October 2012 issue, the American Journal of Evaluation published a study looking at the functioning of advocacy coalitions using social network analysis.  Sure, this kind of quantitative methodology is far from a silver bullet.  However, it may help advocacy evaluators better understand the complexity of their target ecosystem.
  • A very bad mammoth: ‘Cause we all need a good laugh at least once a day.  And who doesn’t enjoy a fantastical story—with witches and a very bad mammoth—as told by a child…in French?  Much of advocacy is about weaving narratives, after all.

Lessons Learned – why I blog: “So What?” aims to contribute to the growing advocacy evaluation field.  While we do feature research findings, innovative methodologies, and evaluation theories every now and then, our blog is far from academic.  I regularly do deep dives into blogs, newspapers and program websites in search of nuggets of wisdom (sometimes mammoth-sized) that our readers may find useful, interesting, and funny too.

Lessons Learned – what I’ve learned: Be brief.  ‘Nuff said.

This winter, we’re continuing our series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I am Melanie Hwalek, CEO of SPEC Associates and a member of AEA’s Cultural Competence Statement Dissemination Core Workgroup. My focus within the Workgroup is to help identify ways to disseminate the Statement and integrate its contents into evaluation policy. AEA’s Think Tank: Adoption of the AEA Public Statement on Cultural Competence in Evaluation: Moving From Policy to Practice and Practice to Policy gave me three big ideas for doing this.

Lesson Learned: Cultural Competence can be in big “P” policy and small “p” policy. Dissemination of the Cultural Competency Statement doesn’t have to start with federal or state level, big “P” policy change. Small polices like setting criteria for acceptable evaluation plans, for assuring that evaluation methods take culture into consideration, and for ensuring culturally sensitive evaluation products can go just as far – or further – in assuring that all evaluations validate the importance of culture in their design, analysis, interpretation and reporting.

Hot Tip: Start where there is a path of least resistance. Agencies that exist to represent or protect minority interests are, themselves, culturally sensitive. These are the agencies that should easily understand the importance of assuring that the evaluations of their programs should include cultural competence. If you are passionate about infusing cultural competence into municipal, state or federal policy, start with these types of agencies since they are likely to understand the importance of culturally sensitive evaluations. Keep in mind, though, that just because an organization “says” it values cultural competence doesn’t mean the really know how to be and act in a culturally competent way.

Hot Tip: Try to go viral. Infusing cultural competence into policy means that we need to be open to all kinds and levels of policy, much of which is identified only through practice. The lesson here is to start promoting cultural competence to anyone and anywhere evaluation planning, methods, analysis and reporting are discussed. In this networked world, the more people who think and talk about cultural competence in evaluation, the more likely it will find its way into evaluation practice and evaluation policy.

Rad resource: William Trochim wrote an informative article on evaluation policy and practice.

This week, we’re diving into issues of Cultural Competence in Evaluation with AEA’s Statement on Cultural Competence in Evaluation Dissemination Working Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Susan Kistler, the American Evaluation Association’s Executive Director and aea365 Saturday contributor. Today I’m writing from Atlanta and the AEA/CDC Summer Evaluation Institute. I wanted to share a bit more about a great resource originally mentioned by Susan Wolfe in her August 2010 post.

Rad Resource – The Community Toolbox: The Community Toolbox provided practical “step-by-step guidance in community building skills.” The Toolbox covers everything from leadership building and group facilitation to developing a strategic plan and organizing for advocacy. Want to review evaluation guidance understandable by a range of stakeholders? Check out the section on Evaluating Community Programs and Initiatives, containing four chapters:

Each chapter is further broken down into multiple sections and each section contains concrete plans, tools and – a unique feature I haven’t seen elsewhere – a PowerPoint summarizing the key points and ready to use to support training and orientation.

Clipped from: ctb.ku.edu (share this clip)

The above represents my own opinions and not necessarily those of the American Evaluation Association. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top