My name is Rahel Wasserfall. I am an internal evaluator and do program development for the International Summer School on Religion in the Public Life (ISSRPL). I’m also affiliated with the Women Studies Research Center at Brandeis University and use mostly qualitative methods, including participant observation and in-depth interviews.
As an evaluator with nine years at the ISSRPL, I’m steeped in my organization’s culture, comfortable with how our programs work and adept at recognizing familiar patterns and behaviors. While these are useful assets, to assess programs effectively, I also need ways to stay “apart.” A key question I’ve been asking is how can internal evaluators keep from falling into analytical ruts? How do we recognize the potential value of information that falls outside the norm or challenges our assumptions? In short, how do we see with “fresh eyes,” when needed?
Lessons Learned: Recognize outliers
- Those within a culture can usually tell what is normative and what falls outside the norm. I suggest paying close attention to the outlier story – information, cases, events and other occurrences that are atypical, when compared to the overall data collected. Instead of dismissing such occurrences, I study them: they may signal a need to dig deeper for more insight.
- A nice example comes from a debriefing following a two-week international and inter-religious program on tolerance. One participant was surprisingly angry with the food: “Why didn’t we have pork?” While not the majority experience, the unexpected comment prompted my review and analysis of participants’ reactions to the food through the years. Ultimately, this yielded broader insights about how our program addressed minorities’ preferences and led to changes.
Hot Tips:
- When collecting qualitative data – for example, through interviews, focus groups, or participant observation – every occurrence, event, and piece of information is potentially useful. Take notes, even if it doesn’t seem to make sense right away.
- Pay special attention to strong emotions and opinions. These are often good clues to investigate further.
- Where possible, use follow-up discussions and debriefings with participants to better understand atypical behaviors and opinions. The added context can sometimes offer surprising insight, which may shift initial interpretations.
- Questioning our assumptions about data can be easier said than done. I find it helpful to do a follow-up review of events, behaviors, and opinions that I’ve decided not to include. Asking why particular data won’t figure in the analysis can help make underlying assumptions explicit and can yield additional, valuable understanding.
Rad Resources:
- Livia Polanyi’s “Stories as Cultural Texts,” in Telling the American Story.
- Gregory Bateson: Steps to an Ecology of the Mind (PDF may load slowly)
The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.