Hello Colleagues! My name is Judith Kallick Russell. I am an independent evaluation consultant in civic engagement, community development and peace building. My clients are national organizations (NGOs) and international organizations (UN agencies, international NGOs and foundations). In my work, I have found that it can be challenging at the report writing stage to provide findings and recommendations which are easily translated into actions for clients. The following are some ideas to address this.

Hot Tip: Include boxes or comments on the side in the findings section which suggest questions for reflection. There may be findings which raise questions you feel require further consideration by the client. Including thought provoking questions or comments in the report – visually separate, but linked to a finding – can encourage the client to explore the issue after the consultancy is completed.

Hot Tip: Frame your recommendations in stages or levels. Some organizations are not ready or able to make big changes at the moment of the evaluation. Once you learn from them what they feel capable or interested in doing, you could structure your recommendations providing options with stages or levels. For example, you might want to describe recommendations for a particular issue according to good, better and best.

Hot Tip: Make time for dialogue when finalizing the report. Consider establishing a process for finalizing the report in the very beginning. You might want to gain informal feedback from a few key stakeholders. Then provide a finalized draft to a representative group within the organization. Maybe conduct a workshop about main findings and recommendations, encouraging participation and collective thinking to deepen their understanding of the issues they face. Incorporate all input into the final report as you see fit. Be sure to focus who you ask input from and what input you are asking for, give clear deadlines, and phrase communications in a way where you are not stuck waiting for someone’s response.

If you want to learn more from Judith, check out the sessions sponsored by the Independent Consulting TIG on the program for Evaluation 2010, November 10-13 in San Antonio. Hope to see you there!

communication, recommendations, reports

I’m Jim Burdine and a few weeks ago I posted a “tip-a-day” (see post here) on presenting community health assessment data back to the community. This is part two of that discussion. Last time I focused on “what to present” this posting focuses on “how to present . . . with results.”

There are two goals for any assessment presentation: (1) reporting the data, and (2) identifying next steps for improving health status. I’ve used a format that is effective in a variety of settings. This assumes, as discussed in my last posting, that the assessment has been a “participatory process.”

Hot Tip: Hold a “regional health summit.” Folks invited include those who have participated in various aspects of the assessment as well as others representing a broad cross section of community sectors. My experience has largely been in medium to smaller communities (500K to 10K population) but participation is roughly the same – about 150-250 folks is a good turnout. The event is split into two parts and can be accomplished in one day or two. The two parts are a presentation of the data, followed by a workshop. Ideally the data presentation is limited to an hour and the actual presentation, or much of it, is by a community member. Avoid the temptation to have more than a couple of speakers – “inclusiveness” is a nice idea but counterproductive in this context. You can invite community members to do “welcomes” and other remarks.

Following the data presentation is another opportunity to develop for community buy-in. In the workshop, we ask folks to select among the “findings” of the assessment (e.g., “access to mental health,” “childhood obesity,” “rural transportation”) and sign up for those topics. Have rooms/tables setup for each group. A convener appointed for each group opens the discussion and is responsible for encouraging folks to commit to participating in an ad hoc task group on this topic that that would agree to meet 2-3 times over the next 2-3 months. At the end of that time (3 months) the larger group is reconvened to hear reports from the various ad hoc task groups and then to endorse a regional health improvement strategy developed at that session (based on the group reports). During the 2-3 month work period, each team reviews pertinent findings and identifies “best practices” of interventions related to that issue. The group then identifies one or two priority activities that incorporate those best practices as adapted to your community.

By following this model, what might have been just a data presentation yields a comprehensive regional strategy with multiple sub-parts that is endorsed by the entire community (at least as represented by the individuals and agencies participating in the overall process).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, participatory

My name is LaMarcus Bolton, and I am the American Evaluation Association’s Technology Director, as well as a doctoral student in Industrial-Organizational Psychology. AEA’s Executive Director, Susan Kistler, is on vacation and I’m filling in with this Saturday’s staff post. Today, I wanted to discuss some of the great strides technology has made in terms of helping to improve communication, in particular focusing on quick, easy, and low-cost/no-cost tools that can help evaluators to keep their costs in check and maintain open communication with stakeholders and clients.

Rad Resource: Despite how far we have come in terms of communication mediums, the good ‘ol telephone is still often the medium of choice. Skype, allows one to make free calls to other Skype users, and has very competitive rates for both national and international calls for non-Skype users. Skype even allows one to create conference calls, although with a limited number of participants. However, if you do not want to rely on voice over Internet protocol (VOIP) technologies like Skype (which, by the way, typically are not as clear as traditional landlines), services such as FreeConferenceCall.com in the US may be worthwhile. I have used FreeConferenceCall.com on a few occasions and have always had pleasurable experiences.

Rad Resource: Although the wide availability of scanners and PDFs have made fax machines less popular, there is sometimes still the need to send or receive a fax. Services like eFax give users a free (though non-local) number that can accept faxes and send them directly to your email address. Unfortunately, to send faxes requires one to upgrade to their eFax Plus service. To address this, services like Fax Zero allows one to upload Microsoft Word and PDF files to send to any fax number in the country. Though these options are reliable, and environmentally friendly, I would caution against using them to send or receive sensitive information.

Rad Resource: If, like myself, you try to stay on the cutting-edge of technology, there have been great strides here as well. In addition to phone calls, Skype also allows for free video calls. This allows one to talk face-to-face with friends, family, and colleagues, by adding a small hint of personalization. Though, for quick and efficient communication, technologies like instant messaging (IM) are king. Though, the problem with instant messaging is that everyone seems to have a different preferred client. For example, popular here within the US are AOL Instant Messenger (AIM) and Google Chat (Gchat), but internationally, Windows Messenger is often preferred. To address this, software like Trillian and Digsby for Windows, and Adium for Mac, allow one to combine all IM services into a single and easy-to-use interface.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

communication, skype

Greetings, I am June Gothberg, the State Technical Assistance Coordinator for the National Secondary Transition Technical Assistance Center (NSTTAC). I am also co-chair of AEA’s Disabilities and Other Vulnerable Populations TIG (DOVP). As we worked together on this week’s AEA365 blog, we wanted bring you practical resources, hints, and tricks for including people with disabilities and other vulnerable populations in your evaluations. DOVP is often approached with simple questions of “how to” in this area. My co-chair, Jennifer will be presenting an AEA Coffee Break Webinar on Thursday on how to include Universal Design in your evaluations and we will be also giving a full hands-on workshop at the AEA Conference in San Antonio with another of our colleagues. We would love to have you there!

In the meantime, I would encourage you to get to know the seven basic principals of Universal Design:

1. Equitable use
2. Flexibility in use
3. Simple and intuitive use
4. Perceptible information
5. Tolerance for error
6. Low physical effort
7. Size and space for approach and use

Hot Tips: Today, I would like to suggest tips and resources from Principal 3: Simple and intuitive use. In other words keep it simple.

• First and foremost, know your participants and prepare. If you know you have people involved who have hearing, vision, ambulatory, sensory challenges, make sure to accommodate ahead of time. Prepare materials and activities with them in mind.
• Remember you may enjoy fancy fonts, backgrounds, and graphics, but the best tools are simple black and white. Besides people with low vision, 8% (10.5 million) of males in the United States are color blind making it difficult to read multicolor materials.
• If are looking to include people with low reading abilities either create the evaluation materials with them in mind or create an alternative assessment at a lower reading level. A simple trick is to use the option provided in your word processing program to rate the ease of readability (MS Word uses the Flesch-Kincaid grade level) and keep your evaluation tool below a 4^th to 6^th grade reading level to include more diverse participation.
• If you are looking to include people in a survey who are either non-verbal or low readers by all means create a survey that uses pictorial responses either in the form of smiles to frowns or actual pictures of activities.
• Offer materials, handouts, and presentations in a variety of formats: large print or Braille if needed.
• Be prepared to include a reader or interpreter.

Rad resources – Websites:

• Universal Design Principles: http://www.design.ncsu.edu/cud/about_ud/udprinciplestext.htm
• Web Content Accessibility Guidelines: http://www.w3.org/WAI/intro/wcag.php
• Presentation Accessibility: http://www.w3.org/WAI/training/accessible

Rad resources – Free picture resources:

• Smiles for surveys: http://www.nsttac.org/products_and_resources/EvaluationToolkit/
• Do to Learn picture resources http://www.dotolearn.com/

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Universal Design

My name is Jennifer Sullivan Sulewski and I am a Research Associate at the Institute for Community Inclusion (ICI) at the University of Massachusetts Boston. At ICI I have worked on a variety of research and evaluation projects related to services and supports for people with disabilities. I currently am most involved with Work Without Limits, a public-private partnership funded by the Massachusetts Medicaid Infrastructure and Comprehensive Employment Opportunities (MI-CEO) grant to strengthen the Massachusetts workforce and advance work opportunities for youth and adults with disabilities in Massachusetts.

Universal Design refers to “the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (http://www.design.ncsu.edu/cud/about_ud/udprincipleshtmlformat.html). The principles of universal design can be applied to evaluation to ensure that all relevant populations are included at every stage of the work, from project design to sharing of findings.

Hot Tip: Universal design is helpful to think about even if your evaluation is not specifically focused on disability programs or issues. You will likely encounter people with disabilities or members of other vulnerable populations whatever your focus, so it is good to be prepared. Moreover, good universal design works better for everyone, even those without disabilities or other barriers. Think of bar patrons watching TV with closed captioning on, or people with strollers or rolling suitcases using elevators and curb cuts.

Hot Tip: One key aspect of applying universal design to evaluation work is to think about all the different ways people communicate or access information. For example, if you are conducting a survey on-line, you might offer the option to do it on paper or over the phone if respondents prefer not to respond by computer. Or if you are doing interviews by phone you may find that some respondents prefer to speak in person or respond by e-mail.

Rad Resource: For more tips, join the AEA Coffee Break Webinar on this topic on Thursday, August 5 at 2PM – click here to learn more and sign up. The Disability and Other Vulnerable Populations TIG is also hosting multiple sessions related to Universal Design, including a skill building workshop and a roundtable, at the upcoming American Evaluation Association Annual Conference in San Antonio the first week in November, so look for us there!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Inclusive, Universal Design

I am American Evaluation Association Executive Director Susan Kistler and I contribute each Saturday’s aea365 post. I have a dirty little secret – I’m obsessed with fonts! Stephanie Evergreen recently gave a great AEA Coffee Break Webinar on Graphic Design for Evaluators (she’ll also be expanding on this topic at AEA’s Annual Conference this November). Stephanie expounded on the importance of color choice and font selection to make great reports and presentations.

Rad Resources: WhatFontIs and WhattheFont both allow you to upload a picture of a font or specify a URL and they will return a best guess at the pictured font. WhatFontIs includes the option to display only free or similar free fonts and then download them on the spot – and it has hundreds in its archive (also browsable). Alternatively, WhattheFont also has a forum where font geeks will help you identify a font if you run into a dead end. Here is a great walk-through of from the MakeUseOf blog. Both of these sites are in beta, and aren’t perfect, but I’ve been impressed with the options that they’ve provided – and the sheer volume of free fonts available from WhatFontIs for free.

Stephanie encouraged her webinar attendees to use kuler to identify particular colors appropriate to a report based on those used by a client. Using tools such as WhatFontIs takes this concept one step further.

Rad Resources: This short article from Chuck Green gives examples of sets of fonts that work together to convey a mood or message. I have no eye for such things and find this guidance invaluable.

Hot Tip: Know your vocabulary when talking about fonts with a designer or using fonts in your word processing program. Here are three definitions that will help you along:

• Serifs: Serifs are the little ‘feet’ that appear on many fonts. Fonts come in two types – Serif fonts such as Times New Roman and Sans serif fonts such as Arial.
• Kerning: Kerning refers to adjusting the space between letters so that the white space is similar from letter to letter, for instance pushing a ‘A ‘and ‘W’ up close to one another producing ‘AW’ so that they actually overlap in vertical space. You can adjust kerning in Microsoft Word under the character spacing. Kerning is used in particular when creating headlines or banners.
• Proportional Typefaces: Almost all typefaces today are proportional, allotting varying amounts of horizontal space to a letter based on its shape so that an ‘l’ receives less space than an ‘m’. Old typewriters used monospace fonts.

Rad Resource: A moment of fun for the font obsessed – take a look at this comic that reflects on font choice http://ow.ly/2hf2e.

Note: These insights are my own and do not necessarily represent the opinion of the American Evaluation Association.

communication, font, presentations

Greetings colleagues. My moniker is Michael Quinn Patton and I do independent evaluation consulting under the name Utilization-Focused Evaluation, which just happens also to be the title of my main evaluation book, now in its 4^th edition. I am a former AEA president. One of the challenges I’ve faced over the years, as many of us do, is making evaluation user-friendly, especially for non-research clients, stakeholders, and audiences. One approach that has worked well for me is using children’s stories. When people come to a meeting to work with or hear from an external evaluator, they may expect to be bored or spoken down to or frightened, but they don’t expect to be read a children’s story. It can be a great ice-breaker to set the tone for interaction.

Hot Tip: I first opened an evaluation meeting with a children’s story when facilitating a stakeholder involvement session with parents and staff for an early childhood/family education program evaluation. The trick is finding the right story for the group you’re working with and the issues that will need to be dealt with in the evaluation.

Rad Resource: Dr. Seuss stories are especially effective. The four short stories in Sneeches and Other Stories are brief and loaded with evaluation metaphors. “What was I scared of?” is about facing something alien and strange — like evaluation, or an EVALUATOR. “Too Many Daves” is about what happens when you don’t make distinctions and explains why we need to distinguish different types of evaluation. “Zaks” is about what happens when people get stuck in their own perspective and can’t see other points of view or negotiate differences. “Sneeches” is about hierarchies and status, and can be used to open up discussions of cultural, gender, ethic, and other stakeholder differences. I use it to tell the story, metaphorically, of the history of the qualitative-quantitative debate.

Hot Tip: Children’s stories are also great training and classroom materials to open up issues, ground those issues in a larger societal and cultural context, and stimulate creativity. Any children’s fairy tale has evaluation messages and implications.

Rad Resource: In the AEA eLibrary I’ve posted a poetic parody entitled “The Snow White Evaluation,” that opens a book I did years ago (1982) entitled Practical Evaluation (Sage, pp. 11-13.) Download it here http://ow.ly/1BgHk.

Hot Tip: What we do as evaluators can be hard to explain. International evaluator Roger Mirada has written a children’s book in which a father and his daughter interact around what an evaluator does. Eva is distressed because she has trouble on career day at school describing what her dad, an evaluator, does. It’s beautifully illustrated and creatively written. I now give a copy to all my clients and it opens up wonderful and fun dialogue about what evaluation is and what evaluators do.

Rad Resource: Eva the Evaluator by Roger Miranda. http://evatheevaluator.com/

communication, Eva, metaphors, stories

My name is Alexey Kuzmin and I am the President of the Process Consulting Company (www.processconsulting.ru) based in Moscow, Russia. As an international evaluation consultant I am interested in exploring new and effective ways of communicating evaluation results to the interested parties. In particular, I often use cartoons. Especially when there is a need to communicate bad news: humor helps.

Rad Resource: Cartoon Resource. This website presents dozens of cartoons for publishing, presentation and promotion. It has a convenient search system that allows you to find cartoons by topic. For instance, there are quite a few consulting cartoons. http://www.cartoonresource.com/consulting_1.htm

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

cartoons, communication, humor

My name is Amy A. Germuth, President of EvalWorks, LLC (http://EvalWorks.com) and owner/blogger at EvalThoughts.com. I’ve worked over the last year on improving my evaluation reports to better meet my client’s needs and have a few great resources to help you do the same.

Rad Resource: “Unlearning Some of our Social Scientist Habits” by Jane Davidson (independent consultant and evaluator extraordinaire, as well as AEA member and TIG leader). http://davidsonconsulting.co.nz/index_files/pubs.htm She recently added some additional thoughts to this work and presented them at AEA’s 2009 annual conference in Orlando. Her PowerPoint slides for this presentation can be found at: http://bit.ly/7RcDso.

Frankly, I think this great article has been overlooked for its valuable contributions. Among other great advice for evaluators (including models or theories but not using them evaluatively and leaping to measurement too quickly), she addresses these common pitfalls when reporting evaluation findings: (1) Not answering (and in some cases not even identifying!) the evaluation questions that guided the methodology, (2) reporting results separately by data type or source, and (3) ordering evaluation report sections like a Master’s thesis. This entertaining article and additional PowerPoint slides really make a case for using the questions that guide the evaluation to guide the report as well.

Rad Resource: The “Evaluation Report Checklist” by Gary Miron (professor at Western Michigan University and former Chief of Staff at The Evaluation Center at WMU) provides a great outline of the eight main sections in an evaluation report (Title page, Exec. Summary, Table of Contents, Introduction and Background, Methodology, Results, Summary and Conclusion, References) and the various things that should be included in each. http://www.wmich.edu/evalctr/checklists/checklistmenu.htm

The author notes that this checklist can be used as a “tool to guide a discussion between evaluators and their clients regarding the preferred contents of evaluation reports and a tool to provide formative feedback to report writers” and can help writers identify the strengths and weaknesses of their report. However, as Gary  notes, evaluation reports differ greatly in terms of purpose, budget, expectations, and needs of the client, thus one may need to consider or weight the checkpoints within sections as well as the relative importance and value of each section when reviewing one’s own writing (or someone else’s).

Using the Evaluation Report Checklist in conjunction with some of Dr. Davidson’s suggestions has increased the quality and utility of my evaluation reports and should do the same for yours.

This contribution is from the aea365 Daily Tips blog, by and for evaluations, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org

checklist, communication, reports