APC TIG Week: Evaluating the effects of NGO organizational change processes on Human Rights advocacy outcomes by Tosca Bruno-van Vijfeijken

I am Tosca Bruno-van Vijfeijken, and I direct the Transnational NGO Initiative at Syracuse University, USA. The Initiative has assisted a number of major international non-governmental organizations (NGOs) to review their leadership and management practices related to large-scale organizational change. My colleagues – Steve Lux, Shreeya Neupane, and Ramesh Singh – and I recently completed an external assessment of Amnesty International’s Global Transition Program (GTP). The assessment objectives included among others the way in which GTP had affected Amnesty’s human rights advocacy outcomes. It also assessed the efficacy of Amnesty’s change leadership and management. One of the fundamental difficulties with this assessment was limitations on time and resources. As such, it was not possible to develop objective measures that directly represented either assessment objectives. Instead, the assessment process primarily triangulated staff perceptions at various levels and coming from across different identity groups within the organization as an approximate measure of the effect of GTP on human rights advocacy goal achievement.

Lessons Learned:

1. The change process was controversial within Amnesty and generated high emotions – both for and against. To protect the credibility of the assessment, we gathered multiple data sources and triangulated staff views through careful sampling for surveying, interviewing and focus group use. A survey with external peers and partners added independent perspectives. Workshops to validate draft findings with audiences that had both legitimacy and diversity of views were critical as well.
2. Evaluating human rights advocacy outcomes is complex. Process and proxy indicators were essential in our assessment.
3. It is equally difficult to attribute human rights advocacy outcomes to Amnesty’s change process, due to the lack of comparative baseline information or counterfactuals.
4. Amnesty is a complex, democratic, membership-based NGO. Given the controversy around the ‘direction of travel’ under GTP, Amnesty promised accountability towards its members by requesting this External Assessment barely four years after the change process had been announced. Statements about the extent of correlation between the GTP and human rights advocacy outcomes thus had to be all the more qualified.

With high profile, high-emotion evaluations like this that are also largely dependent on staff perspectives, the measurement of number of ‘mentions’, and/or recurrent staff views was one obvious indicator. However, as evaluators we also need — in a defensible way — to judge the strength of points made or issues raised – and include not just their frequency but also the gravity of their expression.

5. Evaluators need to be acutely aware of where power is situated in organizations if they want to produce actionable, utilization-focused evaluations.

6. In high profile evaluations such as this, an ability to both understand senior leadership contexts, perspectives and world views and to speak truth to power are important.

Rad Resources: The frameworks by Bolman and Deal (Reframing Organizations: Artistry, Choice and Leadership, 2017) and William and Susan Bridges (Managing Transitions, 2017)  offer consistent value in evaluating organizational change processes in INGOs.

Continue the conversation with us!  Tosca tmbruno@maxwell.syr.edu

The American Evaluation Association is celebrating APC TIG Week with our colleagues in the Advocacy and Policy Change Topical Interest Group. The contributions all this week to aea365 come from our AP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.