AEA365 | A Tip-a-Day by and for Evaluators

TAG | social impact

This is Veronica Olazabal and Nadia Asgaraly of The Rockefeller Foundation’s Measurement and Evaluation team. Welcome to this week’s International and Cross-Cultural Evaluation (ICCE) TIG week where we are providing a sneak preview into the ICCE sessions at AEA 2017 From Learning to Action! The ICCE TIG provides a unique opportunity for us to learn from one another and bridge the gap in conversations happening across geographies, innovations, and communities.

Hot Tip: Trend on the Rise – Financing for International Development

An estimated $2.5 trillion is needed annually to meet the financing gap of the Sustainable Development Goals (SDGs); this makes up more than half of the total investment needs of the developing countries. Among the global development players, such as the United Nations, the private sector is seen as central to filling this gap and achieving the 2030 Agenda for Sustainable Development.

Rad Resource: United Nations, Department of Economic and Social Affairs – Financing for

Development

Newer financing entrants in the global development space come with implications particularly when it comes to standard practices around evaluation of impact. Take for instance the term “social impact measurement” which is readily used among private sector actors such as impact investors, as well as development finance institutions (DFIs) and newer philanthropists to describe the way “impact” is managed, monitored and evaluated. T term accounts for the measurement of both financial and social returns. As newer funders come to the table, we predict that standards of evaluation will swing from largely conventional frameworks around accountability toward standards focused on learning to support decision management.

Hot Tip: Interested in learning more?

This year, the ICCE and SIM TIGs are pleased to co-sponsor the session on Learning From Action: International Perspectives on Social Impact Measurement on Friday, November 10th from 6:30 – 7:15 pm.  Through presentations and discussions, this session will provide an opportunity to share and integrate cross-sectoral lessons from a range of stakeholders involved in this growing space.

Donna Loveridge will look deeper into the intersection of international development evaluation and impact measurement and highlight the implications for the global development industry through research findings from the Donor Committee for Enterprise Development (DCED). Krunoslav Karlovcec, Adviser to the Slovenian Ministry for Economic Development and Technology will discuss his experience implementing a nationally established framework focused on a Social Return on Investment (SROI) approach.  And Alyna Wyatt of Genesis Analytics will present learning from leading the evaluation of innovative financing solutions discussed at the 2017 African Evaluation Association Conference (AFrEA).

Rad Resources: Want to learn more about AEA’s international work?

  • Follow along this week on AEA365’s ICCE TIG sponsored week
  • Check out AEA’s International Working Groups Coffee Break Series here: http://comm.eval.org/coffee_break_webinars/coffeebreak
  • Drop into ICCE’s many AEA Evaluation 2017 Search for International and Cross Cultural Evaluation under “Track” here.

 

The American Evaluation Association is celebrating International and Cross-Cultural (ICCE) TIG Week with our colleagues in the International and Cross-Cultural Topical Interest Group. The contributions all this week to aea365 come from our ICCE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

This is Jane Reisman, Founder of and Senior Advisor to ORS Impact, and an independent social impact advisor. I have taken a deep dive into the practices of impact measurement that are being used by a host of new actors in the areas of innovative finance and market solutions and I encourage your consideration of becoming involved too.

The new investors in social impact across the private sector are working both collaboratively with and independently from traditional social sector actors in the social impact space. Signs point to continued and significant growth of impact investing and other market –based initiatives—particularly as (1) these new industries are beginning to mature, (2) the launch of the post-2015 Sustainable Development Goals (SDGs) which has prompted new discussions about financing for international development and (3) the preferences of women and millennials to align money and meaning.

The growth of innovative finance and market solutions is accompanied by increasing attention to the development of evidence about outcomes and impacts. Yet—there is a gap in the supply of practices, e.g., designs, techniques and tools that are both robust and practical in a market-based context. And evaluators have not been the “go to” profession for these newly arriving social impact actors.

Hot Tip: Both evaluators and impact measurement analysts can benefit by understanding the practices and contexts that guide the each other’s work and contexts.

Rad Resources: The Evaluation Unit of the Rockefeller Foundation is developing a series of papers to explore the implications for evaluation of the new operating environment for social impact work which is increasingly involving market mechanisms in the mix with government, multi- and bi-lateral organizations, philanthropy and non-profit/NGO organizations. See: Streams of Social Impact Work: Building Bridges in a New Evaluation Era with Market-Oriented Players at the Table and the 5th Wave: Social Impact Evaluation.

Hot Tip: The evaluation community faces exciting opportunities to become engaged in bridging the gap between evaluation and impact measurement professions and can play a critical role in supporting the evolution of measurement of outcomes and impacts in a new world order. Look for an AEA coffee break series on this subject this August.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi everyone – My name is Jennifer Novak-Leonard, and I’m a Senior Consultant with WolfBrown, an arts and cultural research and management consultancy.

While many museums conduct regular visitor studies and have evaluators on staff, the idea of having an evaluator in a performing arts organization is largely a foreign concept. Performing arts organizations are in the business of transforming individuals through arts experiences, but evaluation is rarely on their radars and box office receipts and the number of “butts in in seats” are used as proxies of how their art impacts and transforms individual people. Pretty poor proxy measures for impact.

How do you measure the impact of a single artistic performance on its audience? This is the key question Alan Brown, and I explored in Assessing the Intrinsic Impacts of a Live Performance.  This research focused on audience members’ aesthetic experience and the intrinsic impact of the performance, and what an audience member can self-report on a questionnaire shortly after a performance’s conclusion.  Through research since our 2007 study, we and our WolfBrown colleagues have refined impact constructs to be:

  • Captivation: degree to which an individual was engrossed or absorbed
  • Emotional resonance: type of emotional response and the degree of intensity
  • Social bonding and social bridging: sense of connectedness, with respect to self-understanding and identity, and a sense of belonging or pride in one’s community; including appreciation for people different from you.
  • Aesthetic growth and validation: exposure to new or unfamiliar art, artists, or styles of art, or the value derived from seeing familiar work
  • Intellectual stimulation: personal and social dimensions of cognitive engagement

To contextualize impact, we attempt to measure “readiness to receive” the art using context), relevance, and anticipation.

Lessons Learned:

  • Audience members say answering questions about intrinsic impact helps them process and reflect on the art.
  • The value of this approach lies not in data, but in conversations between artistic and administrative staff.
  • Applying these measures to orchestral performances, challenges remain in collecting data on concert programs featuring multiple, different pieces. Some audience members like to report on the (least) favorite, some “average out” their reactions to each piece, while others report on the last piece on the program.

Hot Tips:

  • Comparing staff responses on how they thought a program might affect their audience members with audience data can be eye-opening.
  • Measuring the intrinsic impact of performing arts can sometimes be met with resistance given the highly revered artistic autonomy of artistic directors and staff, and virtuosity of the musicians on stage.  Capturing qualitative data from audience members alongside quantitative data helps ease potential chasms between evaluation and artistic staff.

Rad Resources:

The American Evaluation Association is celebrating Arts, Culture, and Audiences (ACA) TIG Week. The contributions all week come from ACA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

·

I am Phil Halbrook and I would like to share with you information about TRASI – a great tool to help evaluators seeking to measure social impact. The issue of measuring social impact is a hot button one right now and the TRASI database and website is extremely useful.

Rad Resource – TRASI: TRASI is “Tools and Resources for Assessing Social Impact” and it is from the Foundation Center. TRASI has over 150 indexed tools, each with a description of what it does and what entity developed it. Full disclosure – all of the resources there aren’t tools per se, they include best practice guides, methodological guides, and what we might more traditionally consider to be a tool. TRASI also includes a blog and discussion groups where you can discuss the resources available there or broader issues related to assessing social impact.

Rad Resources – Examples from the TRASI Database: So, what’s there?

Campaign Champions Data Collection Tool from the Annie E Casey Foundation: “This is a tool that measures strengthened base of public support. This form tracks and measures the number of “champions” (people who take actions to advance the public will) engaged and the actions these champions undertake as a part of the Born Learning campaign.”

Criteria for Philanthropy at Its Best: “This best practice provides guidelines on values…, effectiveness …, ethics …, and commitment … criteria for evaluating philanthropy.”

Interrupted Time Series Designs [Guide]: “This is an observational method that measures impact of education programs in cases where data before the implementation of the program is available. This method compares the data from before the implementation to the same data afterwards to tease out a trend in achievement.”

And 100+ others

Hot Tip – TRASI on Twitter: You can follow TRASI on Twitter at @FCAtlanta. This apst week, on November 14th, they held their first ever tweet chat on social impact assessment using the hashtag #socimp. If you are on Twitter, the conversation is still going – just search for the hashtag. Not on Twitter? No problem. This link http://hashtags.org/socimp will take you access to a record of the discussion to date with the #socimp hashtag – check it out for insights and resources.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I am David Erickson, the Manager for the Center for Community Development Investments at the Federal Reserve Bank of San Francisco.

In the community development finance and socially-motivated investing worlds, there is both universal agreement for the need for better social outcome measurements and no consensus on how to do it. And yet the pressure to innovate in this area is coming from many sources—from government, consumers, investors, and others.

Lessons Learned: A growing consciousness among consumers and investors about social and environmental issues is already changing the types of products and services that are available in the marketplace. Government, too, is seeking to change the ways it does business by providing more resources to programs that are proven to work and by directing funds away from programs that don’t. Xavier De Souza Briggs, Deputy Director of the Office of Management and Budget, captured this idea at a recent Federal Reserve conference where he explained that leaders in government are trying to change “the DNA of the federal government” so that it can take more risks and reward investments that yield better social outcomes. That change – both in the market and for government – requires better data on social impact.

If it is possible to use investments – by the government and socially-motivated investors – to improve society, the question is how do you know they are succeeding? How do you do this where investments cover a wide range of issues? How do we agree what constitutes impact and what tools can be developed to track it? What is the role government can play to enable standard setting and measurements?

Hot Tip: The Board of Governors of the Federal Reserve System and the Federal Reserve Bank of San Francisco are hosting a meeting titled “Advancing Social Impact Investments through Measurement: New Capital for Community Development” to tackle those questions on March 21, 2011 in Washington, DC.

Resource: Our journal, the Community Development Investment Review, recently devoted the full issue to measuring social impact, including articles exploring social metrics and investing, measuring nonfinancial performance, and the role of the federal government in measuring and defining social impact in the impact investing field. All articles are available for free download.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Archives

To top