Hello! I’m Fiona Remnant, founder of a small UK social enterprise set up to continue developing and applying QuIP – a new qualitative social impact assessment tool, from research conducted in the Centre for Development Studies at the University of Bath.
The QuIP team have been experimenting with a form of ‘goal-free’ evaluation for the past few years, challenging people to think differently about how to collect and use qualitative data, but one of the challenges which keeps us awake at night is how best to convey respondents’ views in a way which captures commissioners’ attention.
We assume and respect that the stories of change we collect represent a ‘truth’ for the ‘intended beneficiaries’ (knowing how contested and hated this phrase is!), so how do we ensure that these complex stories reach decision-makers and challenge assumptions about causation and impact?
Our solution to this has been to rely on our own approach to thematic analysis (within with the wonder of Excel!), counts of which then enable us to plug into software like Tableau and MicroStrategy to visualize findings to answer the following questions:
- What were the key reported drivers of change?
- What were the main outcomes reported?
- How closely did these stories align with the project’s aims; how far can we attribute change to the intervention?
- Putting drivers and outcomes together, what were the causal pathways which led to change?
MicroStrategy Desktop has been a fantastic tool, enabling us to build an interactive dashboard with narrative, filters and different chapter views – avoiding the pitfalls of an over-simplified one-page dashboard and keeping source words up front and centre.
We have created great visualisations for each of these questions…
…but the hardest one to nail was the last; causal pathways – the holy grail! We can produce some very neat network diagrams which show the (often very complex) relationships between drivers and outcomes, but we are still working very hard on how to accurately capture what J.L. Mackie termed INUS in these visualisations; how to label when causes are “insufficient but necessary parts of a condition which is itself unnecessary but sufficient for the result.”
The skill of the analyst in picking up on these links, nuances and complexities cannot be underestimated. Whilst we continue to work on ways to visualize this, we are learning that making it too easy for commissioners to absorb findings ‘at a glance’ risks not doing justice to the rich and complex stories which really speak truth to power.
What evaluators really need to do is draw readers in to engage with the data, be captivated by the words behind the visuals, and acknowledge the complexity behind most causation.
The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.