Hello, my name is Nicole Jackson. I am both an adjunct faculty in the Human Resource Management Certificate program at U.C. Berkeley Extension and a doctoral candidate in Policy, Organization, Measurement, and Evaluation at U.C. Berkeley’s Graduate School of Education. From my previous and current work, I discovered that interviewing is both an art and a science especially when it is used in more formative evaluations. Although considered important, interviews are prone to researcher bias that can impact data collection and reporting. Below I offer some tips to help mitigate forms of research bias during interviews.

Hot Tip #1: Understand how different interview formats may alter findings. The two general categories of interview formats include invidual versus panel interviews and unstructured versus structured interview scripts. Individual or one-on-one interviews as well as unstructured or loose ended-scripts are the most prone to researcher bias. Both of these formats lend easily to loss of control due to different personality types that can affect information collection. Where possible, try to use multiple interviewers or a small panel with a structured interview script to help mitigate and triangulate real-time interview data. Structured interview scripts should always focus on the critical research questions during an evaluation project.

Hot Tip #2: Tailor question types according to personality type and experience level. A variety of question types exist to help evaluators navigate difficult and shy personality types as well as those participants with more or less knowledge and experience. Where possible try to use more open-ended, situational questions with follow-up probes for more shy personalities and those participants with more knowledge and experience. For more difficult personalities, begin with more close-ended (e.g., yes/no) questions and then transition to open-ended question prompts in order to maintain control and focus during the interview.

Hot Tip #3: Never underestimate the role of the interview environment. Nothing can be as frustrating as a distracting interview environment. Always conduct interviews in a quiet, private location with good lighting, appropriate room temperature, and minimum distraction. Have water ready to go to place participants at ease. When using recording technology, always consider Murphy’s Law and have extra notepads and recorders ready on hand. Test all recording equipment during the first two minutes of the interview as a safe-guard.

Hot Tip #4: Be mindful of both verbal and non-verbal language. Experts on interviewing claim that non-verbal communication is just as important as verbal behavior in evaluating the trustworthiness of data. Be aware of how your own body language and those of your participants can alter data collection and assessment. Never use closed poses such as crossed arms while interviewing, which is a sign of defensive behavior. Also, be mindful that non-verbal behaivor is culturally influenced.

Nicole will be conducting a roundtable at evaluation 2010 on improving methods of inquiry to incorporate diverse views and perspectives. Join Nicole and over 2500 colleagues at AEA’s Annual conference this November in San Antonio.

context, interviewing, interviews

Hi, I’m Robert Brunger. I am an evaluator with the Ounce of Prevention Fund of Florida; a Tallahassee based non-profit organization that has worked since 1989 to improve the lives of Florida’s children and families.

If you are planning to use focus groups to learn more about what’s on the minds of your stakeholders, here are some suggestions to help you make sense later out of what gets said during the focus group itself.

Hot Tip #1: Digital recorders really are “the greatest thing since sliced bread!” They are available for less than $40 from electronic retailers. Spend enough to get a model that will allow you to transfer the audio file from the device to your computer. (Get a couple of spare batteries, too!)

Hot Tip #2: Practice with your digital recorder before your use it in a focus group. They are not complicated, but you will want to avoid any undue “fussing” in the focus group setting. Record some practice conversations to get used to the controls and volume levels.

Hot Tip #3: When it gets to “show time,” introduce the digital recorder in a very matter-of-fact fashion, get it started, and then pay no further attention to it until the meeting is over.

Hot Tip #4: Place your recorder in the middle of the table, or on a stool in the middle of a circle of chairs. A recent EVALTALK poster, Daphne LaDue, has made a persuasive case for using two digital recorders, pointed in different directions, as a way to improve your ability to figure out what’s been said later.

Hot Tip #5: Start the digital recorder(s) and a stopwatch at the same time. Your note-taker (and, yes, you do need a note-taker!) can make periodic marginal notes about elapsed time from the stopwatch that can be very helpful later in getting your notes and the recorded audio file(s) to match.

Hot Tip #6: It’s also helpful to create a seating pattern diagram to accompany your notes, and assign everyone an identifier – first names will work well, or numbers, or some uniquely identifying characteristic (e.g., red blouse woman, black man with beard, etc.). You can use this scheme while taking notes to identify individual speakers.

Hot Tip #7: Consider how badly you will need to have a full transcript prepared, as that can be a real “time sink,” taking five to six hours per hour of recorded material. If you are doing multiple groups, or if many people will be involved in interpreting the results, you probably will need them, but for smaller projects, your own summary of what was said, based on your notes and selected quotes from the audio file(s) may be entirely sufficient.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

analysis, digital recorders, focus groups, notes

My name is Jane Davidson and I run an evaluation consulting business called Real Evaluation Ltd. In my work, I advise and support organizations on strategic evaluation; provide evaluation capacity building and professional development; develop tools and templates to help organizations conduct, interpret, and use evaluations themselves; and conduct independent and collaborative evaluations and meta-evaluations.

Over several years’ working with clients and reviewing (at clients’ request) disappointing evaluation reports, I have noticed several critically important elements that make or break evaluation work but are often missing from evaluators’ methodological toolkits.

Hot tip: Clients find it incredibly frustrating to wade through an evaluation report full of evidence and still be none the wiser at the end whether the documented outcomes (let alone the entire program/policy/etc) are any good or not. A key part of an evaluator’s work is to say clearly and explicitly how practically, educationally, socially, or economically (not just statistically) significant outcomes are (severally, and as a set). This is what makes evaluation ‘e-VALU-ation’!

Hot tip: A useful tool for generating real evaluative conclusions is an evaluative rubric. This is a table describing what different levels of performance, value, or effectiveness ‘look like’ in terms of the mix of evidence on each criterion. Grading rubrics have been used for many years in student assessment. Evaluative rubrics make transparent how quality and value are defined and applied. I sometimes refer to rubrics as the antidote to both ‘Rorschach inkblot’ (“You work it out”) and ‘divine judgment’ (“I looked upon it and saw that it was good”)-type evaluations.

Hot tip: Collaborative development of rubrics is a great way to get stakeholders thinking about how ‘quality’ and ‘value’ should be defined for the work they do. It helps build the evaluative thinking needed to generate, understand, accept, and use evaluation findings.

Rad resources:

• Evaluation Methodology Basics: The nuts and bolts of sound evaluation by E. Jane Davidson (2005)
• Improving evaluation questions and answers: Getting actionable answers for real-world decision makers (AEA e-Library’s most viewed and downloaded item)
• Example rubrics in Nunns, Roorda, et al’s (2010) Evaluation of the Recognised Seasonal Employer Policy
• Example rubric (referred to as a ‘global assessment scale’) developed for the evaluation of the Corangamite Salinity Program (case study #10 in Jessica Dart et al’s 1998 Review of Evaluation in Agricultural Extension, pp. 62-63 – a publication from the Rural Industries Research and Development Corporation)
• AEA conference professional development workshop about how to use rubrics (and other evaluation nuts and bolts) to do actionable evaluations (November 10, 2010)
• Strategic evaluation of the workplace assessment program, a relevant recent chapter from Jane in the Handbook of Workplace Assessment (2010)

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Jane? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

Rubric, Toolkit

My name is Susan Kistler. I am the Executive Director of the American Evaluation Association and I contribute each Saturday’s aea365 post.

Resource: Have you ever wanted to hear “riveting talks by remarkable people?” That is the tagline for TED Talks, brought to you by TED, a nonprofit dedicated to “ideas worth spreading.”  TED hosts conferences with some of the world’s best, and most provocative, speakers and then posts those speeches on the web for the world to see for free. Al Gore? Regular speaker. Bill Clinton?  Bill Gates? Jane Goodall? Amy Tan? Yes. Yes. Yes. Yes.

So what does this have to do with evaluation? TED Talks can help you to think ‘out of the box,’ to explore the intersection of art and ideas, to ponder profound issues on which evaluation can have a mitigating effect, to examine our assumptions, and to refine and expand methodologies. Here are three that I have found particularly compelling and that I believe share ideas that can impact practice. Each is free via the link provided.

• Hans Rosling, a Swedish professor of global health, is one of the most repeatedly invited speakers at TED. He has six presentations to date and is one of my favorites. Why? His talks on “The best stats you’ve ever seen,” “Let my dataset change your mindset” and “New insights on poverty” expanded my understanding not only of global health issues, but also regarding how we can convey data so that people will listen and care about what is being said. http://www.ted.com/speakers/hans_rosling.html
• Sheena Iyengar studies how people choose, examining the ways in which personal history, cultural norms, and contextual factors impact ‘free choice’ and even how the concept of free choice is culturally laden. Her TED talk prompted me to purchase her book, The Art of Choosing, and also think more deeply about how choice – and our assumptions about choice – influence evaluation. http://www.ted.com/speakers/sheena_iyengar.html
• Anna Deavere Smith is an actress, playwright, researcher and storyteller. In search of the American character, she interviews people from across the United States and performs excerpts from those interviews – in the interviewee’s own voice and using their words verbatim. Her commentary – on race, equity, justice, optimism – reflected in her TED talk, gains its gravitas and urgency from sharing the authentic voice of the words of stakeholders. http://www.ted.com/speakers/anna_deavere_smith.html

Hot Tip: I saved the best for last. Anna Deavere Smith is going to give the opening keynote at Evaluation 2010 this November in San Antonio where you can learn more from her and over 500 other speakers. Why? Because evaluation is an idea worth spreading.

Anna Deveare Smith, TED

Greetings from Columbia, SC! My name is Heather Bennett, MSW, and I have experience working in the field of evaluation for both the public and private sector. Currently, I work as a Research Associate in the Office of Program Evaluation (OPE) at the University of South Carolina where I have the opportunity to lead and work collaboratively on state and federally funded education initiatives in South Carolina. One of my primary responsibilities is to lead our qualitative data analysis efforts, including the analysis of video or audio recordings of cognitive labs, focus groups, interviews, and responses to open-ended survey items.

Lesson Learned: For my tip-a-day for aea365 I am going to focus on one vital and fundamental lesson I’ve learned through the analysis of responses to open-ended survey items — the quality of the question asked has the greatest impact on the data analysis process.

As evaluators, I’m sure we have all inherited some projects with the corresponding data collection instruments and noticed some issues with the construction of items…or worse, we have looked back on the open-ended items we’ve developed and asked ourselves: “What was I thinking?” Upon inheriting the evaluation of a program, I was soon reminded of the impact item writing can have on data management. Issues of data utility arose as my team and I reviewed the structure of qualitative items and worked to develop clear coding structures for corresponding data.

Hot Tip: Poorly written items do not always follow the “garbage in, garbage out” scenario. However, it takes more time to take-out the trash and get to meaningful data (data cleaning, analysis, coding) when you start with bad items. Below are a few things to keep in mind when developing open-ended items that will support your analysis and coding efforts once the data is collected.

First, you must have a clear understanding of what it is you want to learn about the project before you do anything else. What information do you really hope to gain? What is its utility for the program? This process should be guided by the project scope and involve project stakeholders to ensure the usefulness of the data collected.

Now that you have focused your data collection efforts, use these tips when developing your open-ended item(s):

1. Ask one question at a time.
2. Avoid leading questions.
3. Avoid including personal biases in questions.
4. Be specific about the topic.
5. DO NOT ask questions that can be answered with yes/no.
6. Indicate the number of responses requested from the participant.
7. Ask clear and concise questions to avoid participant fatigue.

Following these tips will serve to improve your efforts in collecting focused and clear information from program participants.

This aea365 Tip-a-Day contribution comes from the American Evaluation Association. If you want to learn more from Heather, join us in San Antonio this November for Evaluation 2010 and check out her session on the Conference Program. If you would like to contribute an aea365 Tip, please send an email to aea365@eval.org.

Data, quality, questions

Hi, my name is Anne Vo and I am a doctoral student in the Graduate School of Education at the University of California, Los Angeles (UCLA). I have been using conversation analysis (CA) to study how program evaluation is taught in informal settings. CA is a research method and tool that was developed during the 1960’s and 1970’s in the field of sociology for the purposes of examining “talk-in-interaction” – the way in which interactions are organized through conversation. It is also used broadly by scholars in disciplines such as anthropology and linguistics. But, evaluators who find that they need to answer questions about the nature and quality of interaction and social relationships in their evaluations may also find CA useful. So, what you will find below are some things to consider while doing CA, a link to an online module for those who may be interested in learning more about what “doing CA” entails, and some references for further exploration.

As with every research and evaluation method, CA has its advantages and disadvantages. In this case, they are quite similar to what we normally encounter when using observational or ethnographic methods. A few methodological considerations, strengths, weaknesses, and pitfalls to consider appear below.

Rad Resource: The following link will take you to an online training module that will walk you through the CA transcription process. This module was created by one of the method’s developers, Dr. Emanuel Schegloff. http://www.sscnet.ucla.edu/soc/faculty/schegloff/TranscriptionProject/index.html

Rad Resources: These provide a great introduction to CA:

• Sacks, H., Schegloff, E.A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4): 696-735.

And, the following books are useful references for those who have long-term interest in CA:

• Ochs, E., Schegloff, E.A. & Thompson, S.A. (Eds.). (1996). Interaction and grammar. New York, NY: Cambridge University Press.
• Schegloff, E.A. (2007). Sequence organization in interaction: A primer in conversation analysis: Volume 1. New York, NY: Cambridge University Press.

Hot Tip: Methodological Considerations.

• Large social constructs that are often studied using CA include human interaction, social relationships, and context.
• In CA, these social constructs are indexed by units of analyses not limited to: turn taking, turn constructional units, and sequence organization.
• “Doing CA” involves meticulous transcription of audio or video recordings using CA notation. Analysis involves concurrent use of the audio/video data with the transcript, but the recordings remain the primary data sources.

Strengths:

• Because CA is inherently a fine-grained tool, one of its strengths is its precision, which can be used to identify nuances that might have otherwise been overlooked.
• It also allows analysis to occur specifically at the conversational level, which is not generally afforded by other methods.

Limitations of the method:

• CA transcription is a time consuming and costly activity.
• And, it requires specialized training in CA methods if it is going to be done well.

Pitfalls:

• The analytic process is sensitive to misinterpretation and over-interpretation so member checking becomes doubly crucial here.
• Likewise, the analyst should be wary of the tendency to essentialize the phenomena that they are observing; rather, she or he should always triangulate these initial patterns before arriving at conclusions.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

conversation analysis

My name is Susan Menkes, and I’m an Applied Developmental Psychology doctoral student at Claremont Graduate University (CGU) and received my Masters in Evaluation from CGU, as well.  My work has focused on applying developmental research to understand effective interview strategies with younger children when evaluating programs serving children and youth.  Below are some tips on how to construct developmentally sensitive questions to increase children’s comprehension and subsequent responses:

Hot Tip: Wh-Questions: Particular question formats, such as wh-question words, are understood before others; words such as, what, which, where, and who are easier to understand than how, when, and why. Understanding which question format is easiest for young children to comprehend is critical to constructing effective questions to elicit the most descriptive output from children.

Hot Tip: Negative Phrases: When trying to elicit evaluative feedback from children, an interviewer may need to phrase the question with a negative (What do you not like about this program/service?).  However, re-phrasing the question to incorporate a semantic negative that is not syntactically negative (What can be done to make this program/service better?), can be useful in eliciting more descriptive detail compared to questions that are syntactically negative.

Hot Tip: Linguistic Complexity: Children may find it particularly difficult to understand linguistically complex questions; simply constructed questions containing one verb are easier for children to comprehend (e.g., ”What do you like to play with your friends when you are outside?” versus “What do you like to play outside with your friends?”). Additional nuances in the basic syntactic words used, even in simple questions, can impact children’s comprehension.

• Pronouns and passive constructions can decrease comprehension.
• Understanding person pronouns (e.g., I, you, and he/she) is difficult for young children; third person pronouns are typically learned later than first and second person forms.
• Passive constructions (e.g., The rabbit is following the cat) are more difficult to comprehend than active constructions (e.g., The rabbit follows the cat).

While including young children throughout the evaluation may not be feasible, they certainly can provide valuable feedback to program managers related to the strengths, weaknesses, or suggested improvements in a program. To maximize the amount of descriptive output children provide to questions, it is critical to construct developmentally sensitive questions that young children can comprehend.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

developmental, qualitative

Hello, my name is Deborah Grodzicki and I just received my Masters in Organizational Behavior and Evaluation from Claremont Graduate University.  I plan to pursue a PhD in Evaluation at UCLA in the fall.  Prior to attending Claremont Graduate University, I investigated complaints against New York City police officers.  During my time as an investigator, I gained experience questioning civilian complainants and police officers about extremely sensitive issues.  Drawing on this experience, I will give some tips on how to obtain essential information without compromising evaluator – stakeholder relationship.

Hot Tip: Do not be a prisoner of your question list. At their most basic, interviews and focus groups consist of the evaluator asking stakeholders a list of questions.  To make these qualitative measures most effective, however, it is critical to maintain flexibility in your questioning and establish a conversational atmosphere.  Do not use the questions as a crutch, but rather as a directional tool for the conversation. Otherwise, you risk casting yourself as an interrogator, which could result in the individual withholding vital information.

Hot Tip: Check your biases at the door. It is natural to come into a situation with personal biases that may affect how you approach an interview and/or focus group. It is important to be mindful of these inevitable biases and make a conscious effort to prevent them from affecting how your questions are phrased and delivered. Faced with a biased or leading question, a stakeholder is more likely to provide more restricted answers that mirror the bias and unduly skew the results.

Hot Tip: Withhold judgment. When conducting interviews and/or focus groups, never give someone the impression that you disapprove of their thoughts, feelings, or actions.  It is up to you as the evaluator to generate a safe, comfortable, and above all, accepting atmosphere. Only then will a stakeholder freely share their impressions about the evaluand.

Hot Tip: Look them in the eye. During my time as an investigator, I was taken aback by how many of my colleagues broke eye contact when a complainant spoke about a sensitive issue. Though seemingly insignificant, this small action can have substantial consequences. Failing to maintain eye contact at the stakeholder’s most vulnerable moment gives the impression that you are uncomfortable hearing what they have to say.  Sensing this can lead the stakeholder to feel self-conscious and promptly shut down.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

focus groups, interviewing, interviews

Hi, my name is René Lavinghouze, I am a senior evaluation scientist in the Office of Smoking and Health at the Centers for Disease Control and Prevention.  Today I will be sharing a tip and a resource concerning storytelling in evaluation.

Hot Tip: Who can resist a picture of a smiling child who is now pain-free because her teeth have been restored and sealed to prevent further decay?  After all, this is the purpose of your program – to change the lives of participants for the better.  Such a simple description of a program’s progress, achievements, or lessons learned is a success story or lesson from the field.  Success stories are relevant to the practice of evaluation and are increasingly used to communicate with stakeholders about a program’s achievements. They are an effective way for prevention programs to highlight program progress as these programs are often unable to demonstrate outcomes for several years.  Therefore, communicating success during program development and implementation is important for building program momentum and sustainability. Success stories come in all shapes and sizes from the 15-second elevator story to the published article.

Rad Resource: A workbook was developed by the Centers for Disease Control and Prevention entitled: Impact and value: Telling your program’s story that focuses on using success stories throughout the program’s life cycle.  This workbook defines success stories, discusses types of success stories, and describes methods for systematically collecting and using success stories to promote your program and influence policy decisions. This workbook is a free download at http://bit.ly/cdcsuccessstories

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

stories

My name is Nina Potter and I am currently the Director of Assessment for the College of Education at San Diego State University. I’d like to share a little about a tool we are using for data visualization.

One of my responsibilities is to work with program directors and department chairs to evaluate academic programs across the college’s eight departments and 30+ programs. Our programs vary greatly in size and each has its own goals and student learning outcomes. Plus, we have some common goals across the college. We wanted to have a common tool that would allow us to share data across the college, but it had to be very flexible in terms of the kinds of data that it could handle as well as the kinds of reports that it could generate. After a lot of exploring, we chose Tableau.

Rad Resource: Before coming to SDSU, I had never heard of Tableau, in fact I had not heard the term “data visualization tool.” First I will tell you what it is NOT. Tableau is not a tool for data entry. You use Tableau to access data from other data sources such as spreadsheets or databases. This was important because our programs use many different tools to collect data, from electronic portfolio systems to paper and pencil tracking (we do require them to at least put the data in a spreadsheet). And, Tableau does not do advanced statistics; although it does do simple regression and t-tests. For statistical tests, we still use other statistic packages.

So what does Tableau do? Tableau allows you to link into multiple data sources, and quickly and easily create interactive graphs and charts that are updated in real time as your data sources are updated. It has a variety of choices for visualizations such as tables, line graphs, bar charts, pie charts and geographical maps. With just a few clicks you can easily change the type of chart, add colors, add filters and drill down to data that fits certain criteria. The charts are interactive so that anyone viewing the charts can apply filters and view the data they want to focus on.

For example, we have some assessments that are given across multiple programs. We can create a chart that looks at student progress over time and add filters such as program, gender, ethnicity, and age. A person who is evaluating the program as a whole can compare the results from program X to program Y to see if there is equity across multiple demographic groups. Additionally, a person who is working with individual students can download a list of students who have failed more than one assessment in a given program.

Want to hear more about Tableau from Nina? Join her on April 29 for “Data in, Brilliance Out with Tableau” as part of AEA’s Coffee Break Demonstration Series. More information and registration may be found at http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx. Free for AEA members!

data visualization, software, technology