AEA365 | A Tip-a-Day by and for Evaluators

TAG | stories

Greetings colleagues. My moniker is Michael Quinn Patton and I do independent evaluation consulting under the name Utilization-Focused Evaluation, which just happens also to be the title of my main evaluation book, now in its 4th edition. I am a former AEA president. One of the challenges I’ve faced over the years, as many of us do, is making evaluation user-friendly, especially for non-research clients, stakeholders, and audiences. One approach that has worked well for me is using children’s stories. When people come to a meeting to work with or hear from an external evaluator, they may expect to be bored or spoken down to or frightened, but they don’t expect to be read a children’s story. It can be a great ice-breaker to set the tone for interaction.

Hot Tip: I first opened an evaluation meeting with a children’s story when facilitating a stakeholder involvement session with parents and staff for an early childhood/family education program evaluation. The trick is finding the right story for the group you’re working with and the issues that will need to be dealt with in the evaluation.

Rad Resource: Dr. Seuss stories are especially effective. The four short stories in Sneeches and Other Stories are brief and loaded with evaluation metaphors. “What was I scared of?” is about facing something alien and strange — like evaluation, or an EVALUATOR. “Too Many Daves” is about what happens when you don’t make distinctions and explains why we need to distinguish different types of evaluation. “Zaks” is about what happens when people get stuck in their own perspective and can’t see other points of view or negotiate differences. “Sneeches” is about hierarchies and status, and can be used to open up discussions of cultural, gender, ethic, and other stakeholder differences. I use it to tell the story, metaphorically, of the history of the qualitative-quantitative debate.

Hot Tip: Children’s stories are also great training and classroom materials to open up issues, ground those issues in a larger societal and cultural context, and stimulate creativity. Any children’s fairy tale has evaluation messages and implications.

Rad Resource: In the AEA eLibrary I’ve posted a poetic parody entitled “The Snow White Evaluation,” that opens a book I did years ago (1982) entitled Practical Evaluation (Sage, pp. 11-13.) Download it here http://ow.ly/1BgHk.

Hot Tip: What we do as evaluators can be hard to explain. International evaluator Roger Mirada has written a children’s book in which a father and his daughter interact around what an evaluator does. Eva is distressed because she has trouble on career day at school describing what her dad, an evaluator, does. It’s beautifully illustrated and creatively written. I now give a copy to all my clients and it opens up wonderful and fun dialogue about what evaluation is and what evaluators do.

Rad Resource: Eva the Evaluatorby Roger Miranda. http://evatheevaluator.com/

The American Evaluation Association is celebrating Best of aea365 week. The contributions all this week are reposts of great aea365 blogs from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Mary Anne Casey here. I’m a Minnesota-based consultant who helps people plan and evaluate programs.  I’ve also done some writing and teaching on qualitative methods, particularly focus groups.

Are you on the lookout for ways to make your presentations more memorable and meaningful?

Rad Resource: Chip and Dan Heath’s book, Made to Stick, is jam-packed with ideas that will make your presentations stand out. Want to read the first chapter for free? Download it at http://www.heathbrothers.com. (Under the resource tab, with lots of other interesting materials.)

Hot Tip: The Heath brothers suggest bringing statistics to life by presenting them in ways that are more human. Here’s an example, taken from their book (p. 144-145):

“Steven Covey, in his book The 8th Habit, describes a poll of 23,000 employees drawn from a number of companies and industries. He reports the poll’s findings:

  • Only 37 percent said they have a clear understanding of what their organization is trying to achieve and why.
  • Only one in five was enthusiastic about their team’s and their organization’s goals.
  • Only one in five said they had a clear “line of sight” between their tasks and their team’s and organization’s goals.
  • Only 15 percent felt that their organization fully enables them to execute key goals.
  • Only 20 percent fully trusted the organization they worked for…”

“Then Covey superimposes a very human metaphor over the statistics. He says, ‘If, say, a soccer team had these same scores, only 4 of the 11 players on the field would know which goal is theirs. Only 2 of 11 would care. Only 2 of the 11 would know what position they play and know exactly what they are supposed to do. And all but 2 players would, in some way, be competing against their own team members rather than the opponent.’”

This soccer analogy humanizes the statistics and makes them more memorable. You can’t help but imagine the chaos on the field if only 4 of the 11 players know which way to go, and only two care. And it is easy to move that image from the soccer team to a work team. It is a great example of how to humanize statistics.

Twin Cities Hot Tip: For the best pho (Vietnamese soup) in town, head over to Ngon, at 799 University Ave W, St Paul, MN.

The American Evaluation Association is celebrating Minnesota Evaluation Association (MN EA) Affiliate Week with our colleagues in the MNEA AEA Affiliate. The contributions all this week to aea365 come from our MNEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Oct/11

3

Susan Eliot on Evaluation Stories

I’m  Susan Eliot, an independent qualitative research and evaluation consultant in Portland Oregon. As an applied researcher, I’m continuously searching for creative yet rigorous qualitative approaches that fit my clients’ budgets and capabilities.

I recently employed a method called the evaluation story to elicit feedback from elementary and middle school students enrolled in an AmeriCorps mentoring program. Good stories are captivating and enduring. They are part of every culture and help us make sense of the world. Storytelling can also be a powerful evaluation tool.

Richard Krueger defines an evaluation story as “a brief narrative account of someone’s experience with a program, event, or activity that is collected using sound research methods.” When used as part of a traditional mixed-method evaluation, Krueger contends we can use stories to help programs and organizations make sense of their programs.

Five key factors differentiate an evaluation story from other stories. According to Krueger, an evaluation story:

  1. Is a deliberate, planned effort using systematic procedures.
  2. Identifies the source of each story.
  3. Verifies stories with the storyteller or others familiar with the story.
  4. Includes a description of how stories were captured/handled using accepted research protocol.
  5. Includes a statement by the evaluator about the degree to which the story represents other individuals with similar circumstances.

Krueger offers the following tips for collecting evaluation stories:

Hot Tip: Using stories to collect evaluation data often takes more time than anticipated. Plan carefully, choose storytellers thoughtfully and engage others in looking out for potential stories.

Hot Tip: To prevent stories from being dismissed as mere anecdotes, describe the process used in obtaining the story, handling the story, verifying the story, and ascertaining its representativeness and meaning.

Hot Tip: It takes skill to get stories out of people. Because it “locks up” people’s memory, avoid asking for a story. Instead ask people to “tell me about . . .”

Hot Tip: Stories usually don’t emerge in perfect form. Expect to invest time in several cycles of editing until the story clearly emerges.

The evaluation story is a great tool for connecting to a program or organization on a personal level. Everyone loves a good story. The trick as an evaluator is to tell a story that resonates with the audience and yet, is grounded in rigorous qualitative methodology. Is it a challenge? It sure is. But the benefits of influencing policy and program development with a well-executed story make it a worthwhile and powerful tool to add to the evaluator’s toolbox.

Rad Resource: Read more about Richard Krueger’s storytelling advice for evaluators on his website, as well as in his chapter on evaluation stories in the Handbook of Practical Program Evaluation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Sharon Smith-Halstead and I would like to share tools for storytelling for evaluators.

Lesson Learned: There is an art to telling stories. Whether you are narrating the story of your grandmother’s journey to America, of your child’s first words, or of the way a program succeeded or failed, there is an art to assembling information so that it provides context, in an authentic way, that is genuine and accurate. It is the art that grabs the attention and moves your reader to action.

There is fiction in the space between
The lines on your page of memories
Write it down but it doesn’t mean
You’re not just telling stories
There is fiction in the space between
You and me

Trace Chapman – Telling Stories

Rad Resource – Impact and Value: Telling Your Program’s Story: This guide from the Centers for Disease Control and Prevention situates storytelling within the broader context of evaluation, noting “While not your main method of presenting data, success stories contribute an effective element to an overall evaluation plan.” Success stories are presented as opportunities for illustrating and celebrating program successes within a more expansive strategy. They provide guidance on soliciting, choosing, and sharing stories.

Lesson Learned: I find that it is rare that a story is fully all success or all failure. Alternatively, we might think of genuine stories as those that truly reflect the nature of an evaluand, its greatness and its weaknesses. However, the celebratory nature, and the power of storytelling, that is shared in the guide, rings true.

The problem is all inside your head
She said to me
The answer is easy if you
Take it logically

She said it’s really not my habit
To intrude
Furthermore, I hope my meaning
Won’t be lost or misconstrued

Paul Simon – 50 ways to Leave Your Lover

Rad Resource – The Story-Dialogue Method: This webpage from the Evaluation Trust focuses on a specific method for employing stories as a data-gathering strategy. It does not get to the point of using the stories in a report, instead focusing on gathering and refining meaningful stores in a way that is authentic and participatory. It details an interactive exercise for eliciting stories in a group setting and then refining those stories, focusing on what, why, so what, and now what questions.

Lesson Learned: Asking people about their stories is powerful. It can be time consuming and mentally exhausting – both for the facilitator and for the participants. Plan breaks and ways to regroup and set aside the project for reflection. I regularly schedule in a 10 minute group walk.

Tell me baby what’s your story
Where do you come from
And where you wanna go this time
You’re so lovely are you lonely
Giving up on the innocence you left behind

Red Hot Chili Peppers – Tell Me Baby

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi, my name is René Lavinghouze, I am a senior evaluation scientist in the Office of Smoking and Health at the Centers for Disease Control and Prevention.  Today I will be sharing a tip and a resource concerning storytelling in evaluation.

Hot Tip: Who can resist a picture of a smiling child who is now pain-free because her teeth have been restored and sealed to prevent further decay?  After all, this is the purpose of your program – to change the lives of participants for the better.  Such a simple description of a program’s progress, achievements, or lessons learned is a success story or lesson from the field.  Success stories are relevant to the practice of evaluation and are increasingly used to communicate with stakeholders about a program’s achievements. They are an effective way for prevention programs to highlight program progress as these programs are often unable to demonstrate outcomes for several years.  Therefore, communicating success during program development and implementation is important for building program momentum and sustainability. Success stories come in all shapes and sizes from the 15-second elevator story to the published article.

Rad Resource: A workbook was developed by the Centers for Disease Control and Prevention entitled: Impact and value: Telling your program’s story that focuses on using success stories throughout the program’s life cycle.  This workbook defines success stories, discusses types of success stories, and describes methods for systematically collecting and using success stories to promote your program and influence policy decisions. This workbook is a free download at http://bit.ly/cdcsuccessstories

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Greetings colleagues. My moniker is Michael Quinn Patton and I do independent evaluation consulting under the name Utilization-Focused Evaluation, which just happens also to be the title of my main evaluation book, now in its 4th edition. I am a former AEA president. One of the challenges I’ve faced over the years, as many of us do, is making evaluation user-friendly, especially for non-research clients, stakeholders, and audiences. One approach that has worked well for me is using children’s stories. When people come to a meeting to work with or hear from an external evaluator, they may expect to be bored or spoken down to or frightened, but they don’t expect to be read a children’s story. It can be a great ice-breaker to set the tone for interaction.

Hot Tip: I first opened an evaluation meeting with a children’s story when facilitating a stakeholder involvement session with parents and staff for an early childhood/family education program evaluation. The trick is finding the right story for the group you’re working with and the issues that will need to be dealt with in the evaluation.

Rad Resource: Dr. Seuss stories are especially effective. The four short stories in Sneeches and Other Stories are brief and loaded with evaluation metaphors. “What was I scared of?” is about facing something alien and strange — like evaluation, or an EVALUATOR. “Too Many Daves” is about what happens when you don’t make distinctions and explains why we need to distinguish different types of evaluation. “Zaks” is about what happens when people get stuck in their own perspective and can’t see other points of view or negotiate differences. “Sneeches” is about hierarchies and status, and can be used to open up discussions of cultural, gender, ethic, and other stakeholder differences. I use it to tell the story, metaphorically, of the history of the qualitative-quantitative debate.

Hot Tip: Children’s stories are also great training and classroom materials to open up issues, ground those issues in a larger societal and cultural context, and stimulate creativity. Any children’s fairy tale has evaluation messages and implications.

Rad Resource: In the AEA eLibrary I’ve posted a poetic parody entitled “The Snow White Evaluation,” that opens a book I did years ago (1982) entitled Practical Evaluation (Sage, pp. 11-13.) Download it here http://ow.ly/1BgHk.

Hot Tip: What we do as evaluators can be hard to explain. International evaluator Roger Mirada has written a children’s book in which a father and his daughter interact around what an evaluator does. Eva is distressed because she has trouble on career day at school describing what her dad, an evaluator, does. It’s beautifully illustrated and creatively written. I now give a copy to all my clients and it opens up wonderful and fun dialogue about what evaluation is and what evaluators do.

Rad Resource: Eva the Evaluator by Roger Miranda. http://evatheevaluator.com/

Rad Resource: Eva the Evaluator by Roger Mirada. http://evatheevaluator.com/

· · ·

Archives

To top