I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor, and I’m thinking deeply this morning. Really deeply. About unanticipated outcomes. It’s fascinating when I think back on what seem like tiny, trivial moments in my life – a sentence spoken, a question asked, an action taken – that had profound impacts on my life and the person I became. This makes me think about what we choose to measure when we engage in evaluation. But it’s not just about me. Unanticipated outcomes have profound effects on many lives.
A friendship that develops in the workplace changes the trajectory of someone’s life. An otherwise unwelcome business in a neighborhood creates jobs and boosts the local economy. A drug originally developed to treat a relatively minor health condition is found to be effective for a much more dangerous one.
Lesson Learned: Unanticipated outcomes have been a consideration in program evaluation for almost as long as evaluation has been a field of inquiry. Marvin Alkin wrote about them as early as 1969: In considering what to measure,
…the evaluator may wish to point out the necessity for broadening the area of concern because of interrelated aspects of the … program, or to consider, as well, various areas of potential unanticipated outcomes. …There are evaluations necessary in providing information during the course of a program about the manner in which the program is functioning, en route objectives are being achieved, and what unanticipated outcomes are being produced. Such information can be of value in modifying the program (program improvement).
Michael Quinn Patton addresses them in his 2015 text, Qualitative Research and Evaluation Methods: Integrating Theory and Practice:
An important contribution of qualitative inquiry is to identify and understand unanticipated consequences, including how unanticipated outcomes intersect with targeted outcomes in program evaluation. This potential of qualitative fieldwork to uncover anticipated outcomes deserves emphasis and is another reason why qualitative studies have become more valued. Statistical studies of outcomes can only imagine what has been thought of, conceptualized, and operationalized in advance. The open and nature of qualitative inquiry, in contrast, is especially useful in capturing unanticipated outcomes. Indeed, one can only turn up side effects, ripple effects, emergent outcomes, and unanticipated impacts through open-ended fieldwork.
Hot Tip: The bottom line is, we may never truly know the reach of programs, interventions, or interactions, so it’s worth considering unanticipated outcomes when we design programs and evaluation plans. How can we prepare ourselves to recognize, document, and evaluate unanticipated outcomes?
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Sheila
Great post.
I have been thinking a lot about both this and external effects, which also tend to get short shrift. An inductive approach involving both program/issue experts and stakeholders could help with this. Mapping the results with network analysis would then make the results tractable and ready for analysis and interpretation.
Happy to discuss anytime, if you are interested.
I don’t know who said it first; I attribute it to Rob Brinkerhoff: “You might not achieve the intended objectives; you will always achieve unintended objectives.”