Hello, my name is Nicole Jackson. I am both an adjunct faculty in the Human Resource Management Certificate program at U.C. Berkeley Extension and a doctoral candidate in Policy, Organization, Measurement, and Evaluation at U.C. Berkeley’s Graduate School of Education. From my previous and current work, I discovered that interviewing is both an art and a science especially when it is used in more formative evaluations. Although considered important, interviews are prone to researcher bias that can impact data collection and reporting. Below I offer some tips to help mitigate forms of research bias during interviews.

Hot Tip #1: Understand how different interview formats may alter findings. The two general categories of interview formats include invidual versus panel interviews and unstructured versus structured interview scripts. Individual or one-on-one interviews as well as unstructured or loose ended-scripts are the most prone to researcher bias. Both of these formats lend easily to loss of control due to different personality types that can affect information collection. Where possible, try to use multiple interviewers or a small panel with a structured interview script to help mitigate and triangulate real-time interview data. Structured interview scripts should always focus on the critical research questions during an evaluation project.

Hot Tip #2: Tailor question types according to personality type and experience level. A variety of question types exist to help evaluators navigate difficult and shy personality types as well as those participants with more or less knowledge and experience. Where possible try to use more open-ended, situational questions with follow-up probes for more shy personalities and those participants with more knowledge and experience. For more difficult personalities, begin with more close-ended (e.g., yes/no) questions and then transition to open-ended question prompts in order to maintain control and focus during the interview.

Hot Tip #3: Never underestimate the role of the interview environment. Nothing can be as frustrating as a distracting interview environment. Always conduct interviews in a quiet, private location with good lighting, appropriate room temperature, and minimum distraction. Have water ready to go to place participants at ease. When using recording technology, always consider Murphy’s Law and have extra notepads and recorders ready on hand. Test all recording equipment during the first two minutes of the interview as a safe-guard.

Hot Tip #4: Be mindful of both verbal and non-verbal language. Experts on interviewing claim that non-verbal communication is just as important as verbal behavior in evaluating the trustworthiness of data. Be aware of how your own body language and those of your participants can alter data collection and assessment. Never use closed poses such as crossed arms while interviewing, which is a sign of defensive behavior. Also, be mindful that non-verbal behaivor is culturally influenced.

Nicole will be conducting a roundtable at evaluation 2010 on improving methods of inquiry to incorporate diverse views and perspectives. Join Nicole and over 2500 colleagues at AEA’s Annual conference this November in San Antonio.

context, interviewing, interviews

My name is Elizabeth Hutchinson and I am a Monitoring and Evaluation Specialist with Land O’Lakes International Development. My work focuses on evaluating United States Government-funded agriculture and food system development programs. Most people in the U.S. know of Land O’Lakes for its branded butter, but the division I work for has 30 years experience implementing international development programs that generate economic growth, improve health and nutrition, and alleviate poverty through market-driven business solutions.

Evaluation activities are integrated throughout the lifecycles of our programs – it is critical for our funders and because ongoing feedback enables us to quickly make programmatic course corrections. We often work with external host-country consultants who bring valuable localized knowledge and evaluation expertise to support our programs. I’d like to share a few lessons learned to improve the practice of working with local evaluators in international settings.

Lesson Learned – The capacity of local consultants varies greatly: Solicit proposals using RFAs from a variety of sources including universities, private organizations, and individuals. Keep in mind that some evaluation skills and expertise may overlap with other fields (i.e. economics, statistics, sociology, public health) which may be beneficial, depending on the evaluation question(s), subject area, and selected methodologies. Reviewing samples of past work (ie. reports, protocols, data sets) and checking references are both good ways to understand the skill sets and capacity of local consultants.

Lesson Learned – International settings can offer unique evaluation challenges: The political/social context in which the evaluation is being conducted may have implications on the quality and timeline. Use knowledgeable local informants to ensure your methodology, questions and timeline are appropriate to the local context. For example, you may not want to conduct household surveys in the month before a significant national election. Allowing a cushion is also critical, particularly if you have strict funder-mandated deadlines. We have found staggering deliverables (drafts, etc) throughout the project timeline to be helpful. There should be ample time built in to make significant changes before a hard/final deadline.

Lesson Learned – Open and ongoing communication is critical: Ensure that expectations around communications are clear and agreed upon upfront. Be explicit about the language you would like deliverables submitted in (i.e. most U.S. organizations likely expect reports in English). The submission/delivery mode should also be determined since sharing documents internationally can be difficult – identify the most appropriate way to share large electronic files, photos, or hardcopy reports, etc. since it can have resource implications. For example, e-mail submission may be more cost- and time-effective than requiring a printed/bound hard copy shipped from the field.

Related Resources:

• IDEAS, International Development Evaluation Association
• RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (SAGE) by Michael Bamberger, Jim Rugh, and Linda Mabry.*

Would you like to discuss issues of capacity building in international development with Elizabeth? She’ll be presenting at Evaluation 2010, the annual conference of the American Evaluation Association this November. Search the online program to find Elizabeth’s roundtable or information about over 500 other sessions.

*AEA members receive 20% off of all books from SAGE when ordered directly from the publisher. If you are a member, sign on to the AEA website and select “Publications Discount Codes” from the “Members Only” menu to access the codes and process.

consultants, funders, international

We are the Green Team, aka Kim Walker and Karen Truesdale, members of the organizing committee for the 2010 Conference of the Canadian Evaluation Society. Our role is to encourage practices that optimize the environmental impact of this large conference.

Why worry about being “green”? …. during a typical five day conference, 500 attendees will use 62,500 plates, 87,500 napkins, 75,000 cups or glasses and 90,000 cans or bottles. Plus there are all the greenhouse emissions from people traveling to and from the conference and paper and plastic waste from conference handouts. (www.meetgreen.com)

There are many online resources that can help and specialists who can offer estimates in advance and both quantitative and qualitative assessments after the event. Here is just one company that provided this service for the 2010 Winter Olympics in Vancouver, BC (www.offsetters.ca).

Rad Resource: Optimizing the environmental impact of a project such as this large conference involves assessing the potential for both negative and positive impacts. We developed a green conference checklist to help us identify where we can make environmentally friendly choices with our venue, programming and other conference activities. We also considered offsite activities, or secondary effects related to the conference – and have given delegates information on what they can do personally to reduce their carbon footprint and support environmentally sustainable practices during their visit (http://www.vanaqua.org/oceanwise/sustainable-seafood.html).

Our environmental evaluation will include input from the attendees via a post-conference survey, interviews with other members of the organizing committee, and observations made during the conference. Our checklist provides the framework for survey questions and observations.

Including an environmental evaluation component to this year’s conference is particularly apt, given focus of our conference – Going Green, Gold and Global: New Horizons for Evaluation (http://c2010.evaluationcanada.ca). What could be more appropriate than to plan for, and then evaluate green initiatives for our conference!

Rad Resource: One of our keynote speakers is Simon Jackson, founder of the Spirit Bear Youth Coalition. Simon was noted as one of Time Magazine’s 60 Heroes of the Planet and he will be speaking on “The Power of One”. We also have a great lineup of workshops and presentations on evaluation in an environmental context, including:

• Water management system in Costa Rica and its impact of water
• Trends in environmental evaluation in Canada, United States and internationally
• Evaluation of the federal leadership in environmental and energy performance for sustainability
• Water and development and the World Bank support
• Impact oriented evaluations system and the environment
• Participants’ motives and means in environmental evaluations
• Bio-economic models of evaluation of fisheries related programs

Hot Tip: We are expecting an inspiring conference. Join us in beautiful Victoria, British Columbia, Canada from May 2 to May 5 to Go Green, Gold, and Global!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Canada, checklist, conference, Environment, Environmental Program Evaluation

Hello, my name is Scott Chaplowe, and I am a Senior M&E Officer with the International Federation Red Cross and Red Crescent Societies (IFRC). I have been working in international monitoring and evaluation for about a decade now, and some of my earliest and most impressionable learning experiences in evaluation were with the AEA at its annual conference. Thus, it is great to see AEA utilize the internet through AE365 and other initiatives for knowledge sharing.

Rad Resources: Field-friendly M&E training and capacity-building modules (Ed. Guy Sharrock). This is a series of nine modules on key aspects of monitoring and evaluation (M&E) for international humanitarian and socioeconomic development programs. They are “field-friendly” in that the module topics were selected to respond to field-identified needs for specific guidance and tools. The intended audience includes managers as well as M&E specialists, and the series can also be used for M&E training and capacity building. The American Red Cross and Catholic Relief Services (CRS) produced the series under their respective USAID/Food for Peace Institutional Capacity Building Grants.

Right now, it seems that the website for Catholic Relief Services is the best location to access the complete series of modules, http://www.crsprogramquality.org/2009/04/fieldfriendly-modules/, and individual module titles include:

• Capacity-Building Guidance
• Monitoring and Evaluation Planning
• Indicator Performance Tracking Tables
• Hiring M&E Staff
• Preparing for an Evaluation
• Managing and Implementing an Evaluation
• Communicating and Reporting on an Evaluation
• Effective Report Writing
• Success and Learning Stories

In addition to the full modules, there are also very handy “Short Cuts” versions of field-friendly M&E training and capacity-building modules (Ed. Guy Sharrock). The Short Cuts provide a ready reference tool for people already familiar with the full modules, or those who want to fast-track particular skills. They can also be reached at Catholic Relief Services, http://www.crsprogramquality.org/2008/09/shortcuts, and individual titles include:

• Capacity-Building Guidance
• Monitoring and Evaluation Planning
• Using Indicator Performance Tracking Tables
• Hiring M&E Staff
• Preparing for an Evaluation
• Managing and Implementing an Evaluation
• Communicating and Reporting on an Evaluation
• Writing Human Interest Stories
• M&E and Ethics

I admit that I am a little biased towards the series as I was a contributing author while working as an M&E Advisor with the American Red Cross’ Tsunami Recovery Program. I wrote the module on Monitoring and Evaluation Planning. The other day a colleague sent me a link for an additional website to directly access this particular module:

http://www.stoptb.org/assets/documents/countries/acsm/ME_Planning_CRS.pdf

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Capacity Building, ethics, hiring, international, M&E, reports, training

Hello, my name is Patricia Rogers. I am Professor of Public Sector Evaluation at the Royal Melbourne Institute of Technology (RMIT University) in Australia and write a blog with Jane Davidson at http://genuineevaluation.com/. I’m particularly interested in evidence-based policy and practice for complicated and complex interventions which require a range of methods and research designs. I often need to communicate across disciplinary and geographic distances, and record and organize useful snippets of information or examples, so I am sharing four rad resources that I find very useful.

Rad resource 1: Delicious social bookmarking http://delicious.com/ It allows you to easily bookmark useful URLs, including pasting in up to 1000 characters of notes, adding tags and sending it to colleagues. You can choose to save it privately or make it accessible to others. You can also search what other people have tagged – for example, using ‘complexity’ or “theory of change”. I have found lots of useful information through this.

Rad resource 2: LiveScribe with Smart Pen. http://www.livescribe.com/ Take notes and record the detailed words at the same time (obviously with permission). Play it back easily by clicking on the words. Upload it to share with colleagues. Particularly useful for those times when in conversation something is explained very clearly and well, and it is great to capture it and be able to go straight back to the specific point. Just make sure you keep track of the very expensive pen!

Rad resource 3: skype. http://www.skype.com/ No cost voice and video calls, easy instant messaging and quick document transfer. I use this a lot for long distance collaboration and it really has made the world a smaller place.

Rad resource 4: World Clock http://www.timeanddate.com/worldclock/ Check the time for an international meeting or set up a customized clock with the cities you need. Invaluable for organizing real-time conversations and teleconferences.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

sharing information, technology