I am Joelle Cook with Organizational Research Services (ORS) in Seattle, Washington. ORS designs, implements and coaches clients in outcome-based planning and evaluation. We specialize in advocacy and policy evaluation and our advocacy-related projects often include the investigation of changes in political will.
Through a demonstration session at the 2011 AEA conference, Steve Mumford and I shared our experiences using the Bellwether Methodology to assess changes in political will for two advocacy-related projects – a pre/post evaluation of a communication campaign promoting library funding to the public and, by extension, local decision makers; and a prospective evaluation of education-reform advocacy efforts. The Bellwether Methodology adds two unique features to basic key informant interviews: 1) the interview sample consists of bellwethers, e.g. thought leaders whose opinions carry substantial weight and predictive value in the policy arena; and 2) interviewees are not informed in advance of the specific policy focus of the interview and instead are told that the interview will discuss a range of policy issues.
Rad Resource: Julia Coffman and Ehren Reed write about the Bellwether Methodology, developed by the Harvard Family Research Project, in their paper: Unique Methods in Advocacy Evaluation. Also check out materials from our AEA presentation in the AEA eLibrary, where we shared experiences adapting and implementing the bellwether methodology and a sample interview protocol we developed.
Hot Tip: Because interviewees are not informed in advance of the interview’s policy focus, evaluators can more objectively assess where the issue is positioned relative to other issues, how decision-makers are thinking and talking about it, how likely decision-makers are to act, and what is realistic progress for the advocacy organization. Prospectively, interviews can inform messaging and communication strategies; retrospectively, they can shed light on the advocacy effort’s contribution to changes in political will.
Hot Tip: Figure out who is in the know about the policy interest but who also track a range of other issues. Bellwethers might be policymakers, media, funders, researchers/think tank staff, business leaders, community leaders or advocates. We worked closely with clients to develop the sample list; however, we had to rely partially on convenience sampling from the list because of bellwethers’ limited availability and turnover in public office.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.