Hello Colleagues! My name is Judith Kallick Russell. I am an independent evaluation consultant in civic engagement, community development and peace building. My clients are national organizations (NGOs) and international organizations (UN agencies, international NGOs and foundations). In my work, I have found that it can be challenging at the report writing stage to provide findings and recommendations which are easily translated into actions for clients. The following are some ideas to address this.

Hot Tip: Include boxes or comments on the side in the findings section which suggest questions for reflection. There may be findings which raise questions you feel require further consideration by the client. Including thought provoking questions or comments in the report – visually separate, but linked to a finding – can encourage the client to explore the issue after the consultancy is completed.

Hot Tip: Frame your recommendations in stages or levels. Some organizations are not ready or able to make big changes at the moment of the evaluation. Once you learn from them what they feel capable or interested in doing, you could structure your recommendations providing options with stages or levels. For example, you might want to describe recommendations for a particular issue according to good, better and best.

Hot Tip: Make time for dialogue when finalizing the report. Consider establishing a process for finalizing the report in the very beginning. You might want to gain informal feedback from a few key stakeholders. Then provide a finalized draft to a representative group within the organization. Maybe conduct a workshop about main findings and recommendations, encouraging participation and collective thinking to deepen their understanding of the issues they face. Incorporate all input into the final report as you see fit. Be sure to focus who you ask input from and what input you are asking for, give clear deadlines, and phrase communications in a way where you are not stuck waiting for someone’s response.

If you want to learn more from Judith, check out the sessions sponsored by the Independent Consulting TIG on the program for Evaluation 2010, November 10-13 in San Antonio. Hope to see you there!

communication, recommendations, reports

My name is Stephanie Evergreen and I work both at The Evaluation Center and at Evergreen Evaluation, LLC. Why am I talking to a group of evaluators about graphic design? What we don’t know can kill our message:

Hot tip: Remember when Office 2007 reduced the default font size from 12 point to 11 point and we all thought that was too small? Well, as it turns out, 12 point font sizes tend to look too big and cartoonish on the printed page. So Office reduced the size, knowing how often we hit the “Print” button. Graphic designers would actually say that between 9 and 11 point font sizes are best for reading evaluation reports. For Web-based dissemination, you can comfortably go a little larger. You wouldn’t want your report to appear comical because of your choices in font.

Hot tip: Speaking of screen readability over print, differences also exist between fonts that can impair readability. Have you ever opened an attachment only to wonder whether the sender had consumed her requisite coffee for the morning because the text is wacky? Fonts don’t always translate across computers. Safe picks for screen reading are Verdana and Georgia. Helvetica and Times Roman are also quite likely to be on most computers and they were designed to be read on the printed page. Pick one or two fonts and keep them consistent across all of your evaluation work with a single client.

Hot tip: Fully justified text looks crisp, professional, and formal, if proper attention can be given to ensure wide gaps don’t appear on certain lines. Left justified text creates a more informal flow, but it is easier to read. Which do you want to communicate to your client? (Hint: Right or center justified text is difficult to read so use it sparingly only for headers or titles of reports.)

Hot tip: Graphic designers tend to disparage bullets as a way to establish emphasis in a written report. Try to highlight your evaluation’s main points with an indent, font change, alignment difference, bold, italic, or even symbols (NOT webdings. Sorry.). Just don’t overdo it. At most, pick two ways of highlighting to combine (i.e., italic indent) and be consistent about it in all of your communications with one client.

Rad Resource: I borrowed these ideas from Ellen Lupton’s (2004) totally awesome, totally accessible book, Thinking With Type. Her website, thinkingwithtype, has visual examples and games to underscore the Hot Tips I listed here, plus lots of others (like how we MUST stop putting two spaces between our sentences).

Want to hear more from Stephanie on Graphic Design for Evaluators? Sign up for her AEA Coffee Break Demonstration Webinar to be held on Thursday, July 22. Learn more on AEA’s CBD page.

Graphic design, reports, type

Hello! It’s Saturday once again. I am Susan Kistler, AEA’s Executive Director, and I contribute each Saturday’s post to aea365.

On April 15, John Nash gave a very well-received webinar for AEA members on Moving Beyond Bullets: Making Presentation Slides Compelling (recorded and archived for member viewing – see also his April 9 aea365 contribution). A key takeaway was the value of incorporating quality, compelling, photos into your presentations. During the Question and Answer portion, attendees asked about where to find free photos for presentations. Below, I am sharing John’s great recommendation (flickr) as well as two others that I have used. These are listed in order of size – but bigger is not always better, it depends on the type of photo you need.

Rad Resource – Stockvault: This free photo site has almost 20,000 photos in its library. Images may only be used for noncommercial purposes, including presentations, noncommercial websites, etc. The photos tend to be of high quality (although there are some less than outstanding ones) and are well indexed. Proportionally, they appear to depict a broad range of racial and ethnic diversity when people are portrayed. This site includes an extensive and varied category of textures and backgrounds that serve as good starting points on which to overlay text in a presentation.

Rad Resource – Stock.xchng: This free photo site has about 400,000 photos in its library. What makes it stand apart is that each one goes through a fairly extensive inspection and review process prior to being accepted for inclusion. The photos are clean and clear and they are highly indexed and thus searchable by keyword and the keywords are both descriptive (such as ‘girl’) and conceptual (such as ‘idea’). The collection can be uneven in terms of coverage with lots of options for one keyword and few for others, but I have had great luck finding the perfect picture here.

Rad Resource – Fickr Creative Commons: This site, a partition of the broader flickr photo sharing site, includes a huge collection of photos that have been tagged with Creative Commons licensing that allows for public use. It can take a few minutes to understand the licensing issues (some allow only for noncommercial use, some allow for no modifications to the photo, some require specialized attribution), but the repository is HUGE with over 100,000,000 – yes 100 million – photos available. One challenge can be that, even with searching, it can be like finding a needle in a haystack. You’ll likely end up sorting through more photos than you would with the other options – but the gems are in there and there are so many from which to choose!

Hot Tip: Always be sure to check out the use and attribution expectations on each site that spell out how you should recognize the photographers and the origin of the photographs.

The above reflects my own work and opinions and not necessarily those of the American Evaluation Association.

communications, photos, presentations, reports

My name is Susan Kistler and I am the Executive Director for the American Evaluation Association. I contribute each Saturday’s post to the aea365 blog. This week I am writing from Atlanta at the Nonprofit Technology conference.

Do you work for or with nonprofit organizations? Have you experienced challenges due to financial constraints that make technology purchases for evaluation beyond the budget?

Hot Tip: Take a look at TechSoup, the “technology place for nonprofits.” TechSoup has resources, training, a peer-to-peer community, and a donated technology program – TechSoup Stock. Their donated tech program gives nonprofits access to products from a range of big name (and not so big name) companies. Examples include the full Microsoft Office Suite including Access and Excel; ArcGIS from ESRI for spatial analysis; and Crystal Reports from SAP for data visualization and reporting. And the cost? Each product has an administrative fee, but most are well below even discounted retail prices. As an example, the full Microsoft Office 2007 suite is $20. Organizations do need to go through a relatively painless qualification process, and the eligibility criteria vary from product to product, but the resource is definitely worth checking out.

The opinions expressed above are my own and do not necessarily represent those of my employer, the American Evaluation Association.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

budget, free, Geographic information systems, GIS, nonprofit, reports, software, technology

My name is John Nash, and I am an associate professor at Iowa State University in the Department of Educational Leadership and Policy Studies with a joint appointment in Human Computer Interaction. I’m also a program strategist, evaluator, and design geek.

Today I’d like to share ways to improve slide presentations.

Hot Tip: Know Your Audience – This is an oft overlooked tip from Nancy Duarte, author of slide:ology, a wonderful book on the art and science of creating great presentations. Duarte suggests seven questions to ask before developing any presentation:

1. What are they like?
2. Why are they here?
3. What keeps them up at night?
4. How can you solve their problem?
5. What do you want them to do?
6. How can you best reach them?
7. How might they resist?

It’s easy to see how these questions would be important to answer in a business or sales presentation. However, amongst evaluators, they are often overlooked when designing a client briefing or conference presentation. I’m especially drawn to question 5, which reminds me that every presentation should be a call to action.

Hot Tip: Let Go of Text – Text can be a crutch for the time-pressed and insecure presenter. Duarte suggests three strategies to excising text as a crutch on your slides:

REDUCE: Practice presenting your slides a few times, then highlight one keyword per bullet point. Deliver your slides from only the keywords, using the rest as notes. Eventually, consider replacing the keyword with an image.

RECORD: Read your presentation out loud and record the audio. Play it back. Once you get over the horror of hearing your own voice, you’ll be able to concentrate on your content and not focus on the slides.

REPEAT: Practice, make note cards, draw a mind map, do anything that helps you visualize or create a cheat sheet. Then, look at your slides and delete as much as possible that’s covered already on your cheat sheet.

Rad Resources: If I could recommend only two books on presenting, they would be the aforementioned slide:ology and Gary Reynold’s Presentation Zen.

Hot Tip: Ignite! Ignite-style presentations are exactly five minutes long using 20 slides that auto-advance every 15 seconds. Using Ignite means delivering the most salient content, from a point of passion, while remaining story-focused (and thus, I argue, more audience focused). For example, watch Molly Wright Steenson’s presentation on the otherwise arcane topic of pneumatic tube networks. Did you adsorb more information than in any other five minutes of your day? Notice how she uses minimal text, good images, and a great story to grab your attention.

Want to learn more from John about giving great presentations? He’ll be offering an AEA Coffee Break Webinar on Moving Beyond Bullets: Making Presentation Slides Compelling on April 15 as part of AEA’s Coffee Break Demonstration Webinar Series (free for AEA members!). Learn more at http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx

facilitation, presentations, proposals, reports

My name is Lija Greenseid. I am a Senior Evaluator, with Professional Data Analysts, Inc. in Minneapolis, MN. We conduct evaluations of stop-smoking programs. Smokers generally have lower education and literacy levels than the general population. Therefore, we want to make sure the materials we develop are understandable to smokers.

Rad Resource: Use a “readability calculator” to check the reading-level of your written materials. I have used this with program registration forms, survey instruments, consent statements, and other materials. Not surprisingly, the first drafts of my materials are often written at a level only grad students (and evaluators) can understand. With a critical eye and a few tweaks I can often rewrite my materials so that they are at an eighth-grade reading level, much more accessible to the people with whom I want to communicate.

A good Readability Calculator can be found here:

http://www.editcentral.com/gwt1/EditCentral.html

It provides you with both a reading ease score, and a number of different measures of the US school grade level of the text.

This blog posting is rated at a high-school reading level. Do you agree?

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

esl, Instruments, Literacy, Readability, reports, Smoking, surveys

Hello, my name is Scott Chaplowe, and I am a Senior M&E Officer with the International Federation Red Cross and Red Crescent Societies (IFRC). I have been working in international monitoring and evaluation for about a decade now, and some of my earliest and most impressionable learning experiences in evaluation were with the AEA at its annual conference. Thus, it is great to see AEA utilize the internet through AE365 and other initiatives for knowledge sharing.

Rad Resources: Field-friendly M&E training and capacity-building modules (Ed. Guy Sharrock). This is a series of nine modules on key aspects of monitoring and evaluation (M&E) for international humanitarian and socioeconomic development programs. They are “field-friendly” in that the module topics were selected to respond to field-identified needs for specific guidance and tools. The intended audience includes managers as well as M&E specialists, and the series can also be used for M&E training and capacity building. The American Red Cross and Catholic Relief Services (CRS) produced the series under their respective USAID/Food for Peace Institutional Capacity Building Grants.

Right now, it seems that the website for Catholic Relief Services is the best location to access the complete series of modules, http://www.crsprogramquality.org/2009/04/fieldfriendly-modules/, and individual module titles include:

• Capacity-Building Guidance
• Monitoring and Evaluation Planning
• Indicator Performance Tracking Tables
• Hiring M&E Staff
• Preparing for an Evaluation
• Managing and Implementing an Evaluation
• Communicating and Reporting on an Evaluation
• Effective Report Writing
• Success and Learning Stories

In addition to the full modules, there are also very handy “Short Cuts” versions of field-friendly M&E training and capacity-building modules (Ed. Guy Sharrock). The Short Cuts provide a ready reference tool for people already familiar with the full modules, or those who want to fast-track particular skills. They can also be reached at Catholic Relief Services, http://www.crsprogramquality.org/2008/09/shortcuts, and individual titles include:

• Capacity-Building Guidance
• Monitoring and Evaluation Planning
• Using Indicator Performance Tracking Tables
• Hiring M&E Staff
• Preparing for an Evaluation
• Managing and Implementing an Evaluation
• Communicating and Reporting on an Evaluation
• Writing Human Interest Stories
• M&E and Ethics

I admit that I am a little biased towards the series as I was a contributing author while working as an M&E Advisor with the American Red Cross’ Tsunami Recovery Program. I wrote the module on Monitoring and Evaluation Planning. The other day a colleague sent me a link for an additional website to directly access this particular module:

http://www.stoptb.org/assets/documents/countries/acsm/ME_Planning_CRS.pdf

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Capacity Building, ethics, hiring, international, M&E, reports, training

My name is Daniela Schröter and I am the director of research at The Evaluation Center and an associate faculty of the Interdisciplinary Ph.D. program in Evaluation (IDPE) at Western Michigan University. In my position, I write a lot of evaluation proposals and implement evaluation projects together with IDPE students and Center staff. One of my pet peeves is making documents produced for potential clients as attractive and user-friendly as possible.

Hot Tip: When using MS Word, make use of the formatting functions. I am especially fond of creating automated Tables of Contents (TOCs) and making extensive use of the options in the “paragraph tab”. Automated TOCs are especially useful when writing long documents. They are easy to update, can help cut down time for editing reports, and hyperlink to the place in the text. You can create similar lists for tables and figures, if needed. The paragraph tab is great for setting up templates where the spacing for different levels of headers and paragraphs is consistently applied. Both functions, in addition to others not discussed here, allow for creating attractive text documents that do not look like they stemmed from an “academic” institution.

Rad Resource: For evaluation proposals, I have been using www.poweredtemplates.com. This site provides templates for MS PowerPoint, brochures, MS Word, and more for free or at cost, depending on the template you choose. These templates are great for proposals and projects with specific themes and can be reused for all kinds of deliverables in a given project (presentations, proposals, management plans, briefing papers, reports, etc.). If you are doing your work within the same discipline and context, you are likely going to find a template that is useful for quite a while and across a lot of projects. The templates are easily adjustable for your needs and adapted across different document types. Further, they look more attractive than the standard templates provided in MS Office.

Rad Resource: Another resource, I am using quite frequently is MS Visio. Visio can be used to create attractive time lines, graphics, logic models, and more. Usually, you can find a version for download and testing at no cost; but once you are hooked, you will likely purchase it. I believe it is the best tool for creating professional graphics. If you are an MS Office user, you have the added benefit of being able to open Visio files within Word to make changes and adjustments to your graphics as needed. On the downside, Visio files may be large and slow down your capacity when working in Word. If you experience such shortcomings, just save your file as a jpeg and insert it in your text as a picture.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

presentations, reports

Kia ora koutou! My name is Jane Davidson and I run an evaluation consulting and capacity building business based in New Zealand. My main clients are NZ central government agencies across multiple sectors (health, education, leadership development, corrections, etc), but I also work with various other kinds of organizations and run evaluation training as well.

Hot Tip: Following on from Jack Mills’ aea365 blog post of 1/7 about project status reports, another that I find very useful is a skeleton report. This is an actual report, minus the content, that is written quite early in the evaluation project showing all the headings/sections and a brief description of which piece of information drops in where. This ensures that (1) the client knows exactly what they are going to get as a deliverable – and if the proposed product doesn’t quite meet their needs we can negotiate that earlier rather than have them disappointed once the report comes in and (2) as the evaluator, I (and anyone I am working with) can see exactly where each piece of evidence is going to fit into the overall puzzle – no more “spare parts” of data that I wasted informants’ (and my) time collecting AND no more “Oops, we don’t have quite enough evidence to answer this question but we have more than we need to answer that one!” I’ve found skeleton reports to be an incredibly useful discipline for making sure that every piece of evidence counts and that I do end up with all that I need to answer the evaluation questions.

Hot Tip: A lot of my reports I structure around the evaluation questions themselves, i.e. one major section for each question. In each section the criteria and evidence are laid out, transparently interpreted, and woven back together to provide a direct, evaluative answer to each question. Similarly, the executive summary consists of 2 pages with 7 +/- 2 short paragraphs, each of which presents one of the 7 +/- 2 overarching evaluation questions followed by a succinct, direct, and explicitly evaluative answer. There’s a bit more about these ideas in an earlier online Journal of MultiDisciplinary Evaluation (JMDE) article and my recent AEA 09 presentation mentioned by Amy Germuth in her aea365 blog post of 1/1. And I’ll talk some more about this, if there’s interest, the week of 17-23 January (US time; that’s 18-24^th for those on the Asia/South Pacific side of the dateline) when I’m online as AEA’s “thought leader” – please join the asynchronous discussion for the week!

As Jane notes, she will be the discussant for the AEA Thought Leaders Series the week of January 17-23 for those who wish to connect with her directly. Learn more and sign up at http://www.eval.org/thought_leaders.asp

reports

My name is Amy A. Germuth, President of EvalWorks, LLC (http://EvalWorks.com) and owner/blogger at EvalThoughts.com. I’ve worked over the last year on improving my evaluation reports to better meet my client’s needs and have a few great resources to help you do the same.

Rad Resource: “Unlearning Some of our Social Scientist Habits” by Jane Davidson (independent consultant and evaluator extraordinaire, as well as AEA member and TIG leader). http://davidsonconsulting.co.nz/index_files/pubs.htm She recently added some additional thoughts to this work and presented them at AEA’s 2009 annual conference in Orlando. Her PowerPoint slides for this presentation can be found at: http://bit.ly/7RcDso.

Frankly, I think this great article has been overlooked for its valuable contributions. Among other great advice for evaluators (including models or theories but not using them evaluatively and leaping to measurement too quickly), she addresses these common pitfalls when reporting evaluation findings: (1) Not answering (and in some cases not even identifying!) the evaluation questions that guided the methodology, (2) reporting results separately by data type or source, and (3) ordering evaluation report sections like a Master’s thesis. This entertaining article and additional PowerPoint slides really make a case for using the questions that guide the evaluation to guide the report as well.

Rad Resource: The “Evaluation Report Checklist” by Gary Miron (professor at Western Michigan University and former Chief of Staff at The Evaluation Center at WMU) provides a great outline of the eight main sections in an evaluation report (Title page, Exec. Summary, Table of Contents, Introduction and Background, Methodology, Results, Summary and Conclusion, References) and the various things that should be included in each. http://www.wmich.edu/evalctr/checklists/checklistmenu.htm

The author notes that this checklist can be used as a “tool to guide a discussion between evaluators and their clients regarding the preferred contents of evaluation reports and a tool to provide formative feedback to report writers” and can help writers identify the strengths and weaknesses of their report. However, as Gary  notes, evaluation reports differ greatly in terms of purpose, budget, expectations, and needs of the client, thus one may need to consider or weight the checkpoints within sections as well as the relative importance and value of each section when reviewing one’s own writing (or someone else’s).

Using the Evaluation Report Checklist in conjunction with some of Dr. Davidson’s suggestions has increased the quality and utility of my evaluation reports and should do the same for yours.

This contribution is from the aea365 Daily Tips blog, by and for evaluations, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org

checklist, communication, reports