AEA365 | A Tip-a-Day by and for Evaluators

TAG | think tank

Hello! My name is Rhonda Williams, and I am a Research and Evaluation Analyst for Region 10 Education Service Center, where I provide internal program evaluations in addition to external research and evaluation services for our partner districts. As a new evaluator, I have run into times where the next steps are not always clear, or I have various paths to analyze the data provided to me where I need to make the executive decisions of how to proceed. During these times I have found these three tips to help me refocus and obtain clarity.

Hot Tip #1: Read research or evaluations about unrelated topics

Since I am in the education field, I often find large think tanks or foundations to learn about issues in transportation, healthcare or urban-renewal. I often subscribe to some of the organization’s newsletters in order to begin receiving their information and research. I always learn new ways to design research projects, to present data, to analyze data, to present my findings and even craft my final reports. I gain inspiration and perspective by consuming the hard work of others!

Rad Resource: Link of US Think Tanks

Hot Tip #2: Reach out of my local peers

My local area has an emerging group of evaluators primarily in education who meet each month; I have met many who are involved in similar work to myself in other organizations. I am now able to reach out via phone to thought-partner or schedule coffee-dates to gain assistance in my work. Using my local professional network allows me real-time access to others and allows others to access me as well. These pro-bono consultancies help me to move my work forward, in addition, to assist in building my professional network.

Hot Tip #3: Re-read previous reports, projects, and even dissertations.

The process of understanding and interfacing with my old thinking reminds me that I am capable of creating and executing strategies to answer complex research questions. It also connects me to the present problems of practice by providing alternative ways of conceptualizing and analyzing the problem. I am also able to revisit the different frameworks and research designs that have been successful for me in the past.

I have come to use these tips quite often, and it keeps me connected to my work in ways I did not initially imagine. Consuming the work of other researchers across industries and even continents allow me to take the limits off of your projects and explore other opportunities to examine, re-examine and move forward in your work.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Dena Lomofsky, managing member of Southern Hemisphere and Vanesa Weyrauch, co-founder of Politics & Ideas work together to deliver specialist online courses on MEL for think tanks, in the field of policy influence.

There are many exciting ways of doing MEL for think tanks, which we would love to share with you. The complexity of policy influence has been long acknowledged: still organisations are often asked to prove attribution or direct contribution to a specific policy change. Thus, how can we use MEL to bridge this gap between uncertainty and complexity and the need to prove we are having impact with the resources we invest?

Typically think tank staff is overstretched; meeting multiple demands of producing high quality research evidence, disseminating results and frequently securing funds as well! As an evaluation respondent once commented: “We appear to be on top of it, under the surface we a paddling like crazy.” Is MEL of policy influence possible and worth the effort?

Hot Tips:

You define success: Set your own policy influence goals, based on your understanding of the policy issue, process and context. Clearly defining your policy influence objectives is important: is it just about affecting concrete policy content or is your major battle to convince policymakers about a new way of framing a policy solution? Clear policy influence objectives will also lead to better evaluation questions.

Keep it simple: Increase the sophistication of your MEL effort as the confidence of your organisation grows.

Build on your strengths: MEL requires research skills. Isn’t this skill inherent in most think tanks? Conduct a situation analysis, identify how you could harness the skills and data you have to inform program improvement.

lemovsky

Choose a suitable MEL framework: There are many frameworks for MEL that also influence your choice of evaluation technique; get to know what is out there are find one that suits your organisational capacities and approach.

Lessons learned: Use internal reflection to motivate people about MEL, otherwise it can become an administrative chore. After action review is a good way to engage people in evaluative learning practices.

The onthinktanks school online course (9/26 – 11/11) on MEL for policy influence explores these topics. The fee is $500.

Rad Resources:

A framework for designing MEL for policy research projects: Pasanen, T., and Shaxson, L. (2016) ‘How to design a monitoring and evaluation framework for a policy research project’. ODI

Onthinktanks and Politics&Ideas collection of articles on MEL for policy influence.

A literature review on evidence to decision making that provides a framework for theory based evaluations: Langer L, Tripney J, Gough D (2016). The Science of Using Science: Researching the Use of Research Evidence in Decision-Making. London: EPPI-Centre, UCL

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi! I’m Johanna Morariu, Senior Associate with Innovation Network. Innovation Network (http://www.innonet.org/) is an evaluation consulting firm that provides evaluation consulting services to nonprofits and funders, and works to build the sector’s evaluation capacity. I lead Innovation Network’s environmental evaluation work, with a strong concentration on environmental advocacy evaluation.

A few months ago we began working with the Post Carbon Institute to develop an organizational evaluation framework. The Post Carbon Institute provides individuals, communities, businesses, and governments with the resources needed to understand and respond to the interrelated economic, energy, environmental, and equity crises that define the 21st century. As the organization identifies with being both a think tank organization and an advocacy organization, the evaluation team knew both elements needed to be present in the evaluation framework.

Since Innovation Network has been at the forefront of the advocacy evaluation field development, over the years we have amassed ample advocacy evaluation resources (free log in required). Early on we identified a need to supplement our existing expertise with a field scan of think tank evaluation literature. We found four resources that were particularly helpful, and we synthesized the resources to create a matrix of assessment areas.

Rad Resources:

  • Donald E. Abelson (2010). Is Anybody Listening? Assessing the Influence of Think Tanks. Chapter 1 in the edited volume, Think Tanks and Public Policies in Latin America.
  • Richard Bumgarner, Douglas Hattaway, Geoffery Lamb, James G. McGann, and Holly Wise (2006). Center for Global Development: Evaluation of Impact. Arabella Philanthropic Investment Advisors, LLC for the Bill & Melinda Gates Foundation, the William and Flora Hewlett Foundation, the John D. and Catherine T. MacArthur Foundation, and the Rockefeller Foundation.
  • Ingie Hovland (2007). Making a Difference: M&E of Policy Research. Working paper 281 for the Overseas Development Institute, London, UK.
  • James G. McGann (2006). Best Practices for Funding and Evaluating Think Tanks & Policy Research. McGann Associates for the William and Flora Hewlett Foundation.

The next step was to fine tune the areas of assessment for think tank evaluation and advocacy evaluation with interview data collected from key informants. Then, in consultation with key staff from the Post Carbon Institute, we created an organizational theory of change. The theory contains information about the organization’s mission; audiences; strategies; focusing events, crises, and windows of opportunity; desired shifts, and impact.

From the theory of change, the evaluators developed corresponding outcomes, indicators, and data collection tools, designed to provide actionable information about organizational strategy. A three tier approach to data collection was recommended—ongoing (e.g., meeting tracking, media tracking, champion tracking), annual (e.g., Bellwether interviews, partner survey, capacity assessment), and as needed (e.g., Intense Period Debrief).

The American Evaluation Association is celebrating Earthweek with our colleagues in the Environmental Program Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EPE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Archives

To top