AEA365 | A Tip-a-Day by and for Evaluators

TAG | notes

My name is Rahel Wasserfall. I am an internal evaluator and do program development for the International Summer School on Religion in the Public Life (ISSRPL). I’m also affiliated with the Women Studies Research Center at Brandeis University and use mostly qualitative methods, including participant observation and in-depth interviews.

As an evaluator with nine years at the ISSRPL, I’m steeped in my organization’s culture, comfortable with how our programs work and adept at recognizing familiar patterns and behaviors. While these are useful assets, to assess programs effectively, I also need ways to stay “apart.” A key question I’ve been asking is how can internal evaluators keep from falling into analytical ruts? How do we recognize the potential value of information that falls outside the norm or challenges our assumptions? In short, how do we see with “fresh eyes,” when needed?

Lessons Learned: Recognize outliers

  • Those within a culture can usually tell what is normative and what falls outside the norm. I suggest paying close attention to the outlier story – information, cases, events and other occurrences that are atypical, when compared to the overall data collected.  Instead of dismissing such occurrences, I study them: they may signal a need to dig deeper for more insight.
  • A nice example comes from a debriefing following a two-week international and inter-religious program on tolerance. One participant was surprisingly angry with the food: “Why didn’t we have pork?” While not the majority experience, the unexpected comment prompted my review and analysis of participants’ reactions to the food through the years. Ultimately, this yielded broader insights about how our program addressed minorities’ preferences and led to changes.

Hot Tips:

  • When collecting qualitative data – for example, through interviews, focus groups, or participant observation – every occurrence, event, and piece of information is potentially useful. Take notes, even if it doesn’t seem to make sense right away.
  • Pay special attention to strong emotions and opinions. These are often good clues to investigate further.
  • Where possible, use follow-up discussions and debriefings with participants to better understand atypical behaviors and opinions. The added context can sometimes offer surprising insight, which may shift initial interpretations.
  • Questioning our assumptions about data can be easier said than done. I find it helpful to do a follow-up review of events, behaviors, and opinions that I’ve decided not to include.  Asking why particular data won’t figure in the analysis can help make underlying assumptions explicit and can yield additional, valuable understanding.

Rad Resources:

The American Evaluation Association is celebrating Internal Evaluators TIG Week. The contributions all week come from IE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· ·

I am Karen Chance and I learned the Cornell Note Taking System (CNTS) in a ‘how to succeed’ in college course my freshman year. I found it useful throughout college and have used a modified version of it since then as well when taking interview notes. I was surprised when I mentioned it to friends and found out that they hadn’t heard of it.

Hot Tip: The CNTS consists of dividing your page into three sections. You take your notes in the largest section, add cues or questions related to those notes in the left column, and add a reflective summary at the bottom. Here is a one page pdf showing the organization of the page and what you are putting in each section.

Hot Tip: When using the CNTS for classes, I found it useful to study (as they suggest) using the cues (keywords or questions) on the left with the actual notes covered. When I was confident that I knew something, I’d add a small check to the upper left hand of the cue so that I could focus in on content I was less solid on during the next review.

Hot Tip: I started using the CNTS for interviews only because I had gotten so used to organizing my note-taking in this way. Although I didn’t have to go back and review for a test, I found that I could still put the keywords in the left column and use these to more readily find commonalities for qualitative analysis. I kept up the practice of adding a summary at the bottom after I  had completed an interview (although I will admit that I did not add a summary to every page), so that I could restate and highlight what I had heard, and in particular so that I could note any connections to other interviews for later cross-reference.

Rad Resource: On this page you can create your own notepaper that is pre-formatted for CNTS use.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi, I’m Robert Brunger. I am an evaluator with the Ounce of Prevention Fund of Florida; a Tallahassee based non-profit organization that has worked since 1989 to improve the lives of Florida’s children and families.

If you are planning to use focus groups to learn more about what’s on the minds of your stakeholders, here are some suggestions to help you make sense later out of what gets said during the focus group itself.

Hot Tip #1: Digital recorders really are “the greatest thing since sliced bread!” They are available for less than $40 from electronic retailers. Spend enough to get a model that will allow you to transfer the audio file from the device to your computer. (Get a couple of spare batteries, too!)

Hot Tip #2: Practice with your digital recorder before your use it in a focus group. They are not complicated, but you will want to avoid any undue “fussing” in the focus group setting. Record some practice conversations to get used to the controls and volume levels.

Hot Tip #3: When it gets to “show time,” introduce the digital recorder in a very matter-of-fact fashion, get it started, and then pay no further attention to it until the meeting is over.

Hot Tip #4: Place your recorder in the middle of the table, or on a stool in the middle of a circle of chairs. A recent EVALTALK poster, Daphne LaDue, has made a persuasive case for using two digital recorders, pointed in different directions, as a way to improve your ability to figure out what’s been said later.

Hot Tip #5: Start the digital recorder(s) and a stopwatch at the same time. Your note-taker (and, yes, you do need a note-taker!) can make periodic marginal notes about elapsed time from the stopwatch that can be very helpful later in getting your notes and the recorded audio file(s) to match.

Hot Tip #6: It’s also helpful to create a seating pattern diagram to accompany your notes, and assign everyone an identifier – first names will work well, or numbers, or some uniquely identifying characteristic (e.g., red blouse woman, black man with beard, etc.). You can use this scheme while taking notes to identify individual speakers.

Hot Tip #7: Consider how badly you will need to have a full transcript prepared, as that can be a real “time sink,” taking five to six hours per hour of recorded material. If you are doing multiple groups, or if many people will be involved in interpreting the results, you probably will need them, but for smaller projects, your own summary of what was said, based on your notes and selected quotes from the audio file(s) may be entirely sufficient.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to

· · ·


To top