My name is Jane Davidson and I run an evaluation consulting business called Real Evaluation Ltd. In my work, I advise and support organizations on strategic evaluation; provide evaluation capacity building and professional development; develop tools and templates to help organizations conduct, interpret, and use evaluations themselves; and conduct independent and collaborative evaluations and meta-evaluations.

Over several years’ working with clients and reviewing (at clients’ request) disappointing evaluation reports, I have noticed several critically important elements that make or break evaluation work but are often missing from evaluators’ methodological toolkits.

Hot tip: Clients find it incredibly frustrating to wade through an evaluation report full of evidence and still be none the wiser at the end whether the documented outcomes (let alone the entire program/policy/etc) are any good or not. A key part of an evaluator’s work is to say clearly and explicitly how practically, educationally, socially, or economically (not just statistically) significant outcomes are (severally, and as a set). This is what makes evaluation ‘e-VALU-ation’!

Hot tip: A useful tool for generating real evaluative conclusions is an evaluative rubric. This is a table describing what different levels of performance, value, or effectiveness ‘look like’ in terms of the mix of evidence on each criterion. Grading rubrics have been used for many years in student assessment. Evaluative rubrics make transparent how quality and value are defined and applied. I sometimes refer to rubrics as the antidote to both ‘Rorschach inkblot’ (“You work it out”) and ‘divine judgment’ (“I looked upon it and saw that it was good”)-type evaluations.

Hot tip: Collaborative development of rubrics is a great way to get stakeholders thinking about how ‘quality’ and ‘value’ should be defined for the work they do. It helps build the evaluative thinking needed to generate, understand, accept, and use evaluation findings.

Rad resources:

• Evaluation Methodology Basics: The nuts and bolts of sound evaluation by E. Jane Davidson (2005)
• Improving evaluation questions and answers: Getting actionable answers for real-world decision makers (AEA e-Library’s most viewed and downloaded item)
• Example rubrics in Nunns, Roorda, et al’s (2010) Evaluation of the Recognised Seasonal Employer Policy
• Example rubric (referred to as a ‘global assessment scale’) developed for the evaluation of the Corangamite Salinity Program (case study #10 in Jessica Dart et al’s 1998 Review of Evaluation in Agricultural Extension, pp. 62-63 – a publication from the Rural Industries Research and Development Corporation)
• AEA conference professional development workshop about how to use rubrics (and other evaluation nuts and bolts) to do actionable evaluations (November 10, 2010)
• Strategic evaluation of the workplace assessment program, a relevant recent chapter from Jane in the Handbook of Workplace Assessment (2010)

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Jane? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

Rubric, Toolkit

My name is Susan Kistler. I am the Executive Director of the American Evaluation Association and I contribute each Saturday’s aea365 post.

Resource: Have you ever wanted to hear “riveting talks by remarkable people?” That is the tagline for TED Talks, brought to you by TED, a nonprofit dedicated to “ideas worth spreading.”  TED hosts conferences with some of the world’s best, and most provocative, speakers and then posts those speeches on the web for the world to see for free. Al Gore? Regular speaker. Bill Clinton?  Bill Gates? Jane Goodall? Amy Tan? Yes. Yes. Yes. Yes.

So what does this have to do with evaluation? TED Talks can help you to think ‘out of the box,’ to explore the intersection of art and ideas, to ponder profound issues on which evaluation can have a mitigating effect, to examine our assumptions, and to refine and expand methodologies. Here are three that I have found particularly compelling and that I believe share ideas that can impact practice. Each is free via the link provided.

• Hans Rosling, a Swedish professor of global health, is one of the most repeatedly invited speakers at TED. He has six presentations to date and is one of my favorites. Why? His talks on “The best stats you’ve ever seen,” “Let my dataset change your mindset” and “New insights on poverty” expanded my understanding not only of global health issues, but also regarding how we can convey data so that people will listen and care about what is being said. http://www.ted.com/speakers/hans_rosling.html
• Sheena Iyengar studies how people choose, examining the ways in which personal history, cultural norms, and contextual factors impact ‘free choice’ and even how the concept of free choice is culturally laden. Her TED talk prompted me to purchase her book, The Art of Choosing, and also think more deeply about how choice – and our assumptions about choice – influence evaluation. http://www.ted.com/speakers/sheena_iyengar.html
• Anna Deavere Smith is an actress, playwright, researcher and storyteller. In search of the American character, she interviews people from across the United States and performs excerpts from those interviews – in the interviewee’s own voice and using their words verbatim. Her commentary – on race, equity, justice, optimism – reflected in her TED talk, gains its gravitas and urgency from sharing the authentic voice of the words of stakeholders. http://www.ted.com/speakers/anna_deavere_smith.html

Hot Tip: I saved the best for last. Anna Deavere Smith is going to give the opening keynote at Evaluation 2010 this November in San Antonio where you can learn more from her and over 500 other speakers. Why? Because evaluation is an idea worth spreading.

Anna Deveare Smith, TED

My name is Michael Quinn Patton and I am an independent evaluation consultant. That means I make my living meeting my clients’ information needs. Over the last few years, I have found increasing demand for innovative evaluation approaches to evaluate innovations. In other words, social innovators and funders of innovative initiatives want and need an evaluation approach that they perceive to be a good match with the nature and scope of innovations they are attempting.  Out of working with these social innovators emerged an approach I’ve called developmental evaluation that applies complexity concepts to enhance innovation and support evaluation use.

Hot Tip: Innovations are different from standard projects and programs.  Innovators are often different from people implementing typical programs.  Innovators are in a hurry, value rapid, real time feedback, have a high tolerance for ambiguity, embrace uncertainty, learn quickly, and adapt rapidly to changed conditions. They’re not always sure where they’re heading, so they resist being boxed in by concrete, pre-set targets. They’re propelled into action more by vision than by clear, specific and measurable outcomes. They want an evaluation approach attuned to their fast pace and innovative spirit. They are at home in complex dynamic systems. Such systems characterize the world in which they live and work. Thus, they want an evaluation approach attuned to complexity.

Hot tip: Complex situations challenge traditional evaluation practices. Complexity can be defined as situations in which how to achieve desired results is not known (high uncertainty), key stakeholders disagree about what to do and how to do it, and many factors are interacting in a dynamic environment that undermine efforts at control, making predictions and static models problematic.  Complexity concepts include nonlinearity (small actions can produce large reactions), emergence (patterns emerge from self-organization among interacting agents), and dynamic adaptations (interacting elements and agents respond and adapt to each other).

Hot tip: Developmental evaluation aims to meet the needs of social innovators by applying complexity concepts to enhance innovation and use. Developmental evaluation focuses on what is being developed through innovative engagement.

Rad Resources:
•    Developmental evaluation: Applying complexity concepts to enhance innovation and use by Michael Quinn Patton (Guilford Press, 2010).*
•    A developmental evaluation primer. Jamie Gamble. (2008). Montréal: The J.W. McConnell Family Foundation.
•     DE 201: A Practitioner’s Guide to Developmental Evaluation by Elizabeth Dozois,
Marc Langlois and Natasha Blanchet-Cohen. Montréal: The J.W. McConnell Family Foundation.
•    AEA Annual Conference professional development workshop on Developmental Evaluation, with Michael Quinn Patton, November 8-9, San Antonio.

*AEA Members receive 20% off on all books ordered directly from Guilford. If you are a member, sign into the AEA website at http://eval.org/ and select “Publications Discount Codes” from the “Members Only” menu to access the discount codes and process.

developmental evaluation, funders, innovation

My name is Sandra Eames, and I am a faculty member at Austin Community College and an independent evaluation consultant.

For the last several years, I have been the lead evaluator on two projects from completely different disciplines.  One of the programs is an urban career and technical education program and the other is an underage drinking prevention initiative.  Both programs are grant funded, yet; they require very different evaluation strategies because of the reportable measures that the funding source requires.  Despite the obvious differences within these two programs’ such as deliverables and target population, they still have similar evaluation properties and needs. The evaluation design for both initiatives was based on a utilization-focused (UF) approach which has universal applicability because it promotes the theory that program evaluation should make an impact that empowers stakeholders to make data grounded choices (Patton, 1997).

Hot Tip: UF evaluators want their work to be useful for program improvement, and increase the chances of stakeholders utilizing their data-driven recommendations.  Following the UF approach could avoid the chance of your work going on a shelf or in a drawer somewhere.  Including stakeholders in the early decision making steps is crucial to this approach.

Hot Tip: Begin a partnership with your client early on that will lay the groundwork for a participatory relationship and it is this type of relationship that will ensure that the stakeholder utilizes the evaluation. What good has all your hard work done if your recommendations are not used for future decision-making? This style helps to get buy-in which is needed in the evaluation’s early stages.  Learn as much as you can about the subject and intervention that they are proposing and be flexible.  Joining early can often prevent wasted time and efforts especially if the client wants feedback on the intervention before they begin implementation.

Hot Tip: Quiz the client early as to what they do and do not want evaluated, help them to determine priorities especially if they are under a tight budget or short on time for implementation of strategies.  Part of your job as evaluator is to educate the client on the steps that are needed to plan a useful evaluation. Inform the client that you report all findings both good and bad upfront might prevent some confusion come final report time.  I have had a number of clients who thought that the final report should only include the positive findings and that the negative findings should go to the place were negative findings live.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources! And, if you want to learn more from Sandra, check out the CAP Sponsored Sessions on the program for Evaluation 2010, November 10-13 in San Antonio.

design, stakeholders, utilization-focused evaluation

My name is Stephanie Evergreen and I work both at The Evaluation Center and at Evergreen Evaluation, LLC. Why am I talking to a group of evaluators about graphic design? What we don’t know can kill our message:

Hot tip: Remember when Office 2007 reduced the default font size from 12 point to 11 point and we all thought that was too small? Well, as it turns out, 12 point font sizes tend to look too big and cartoonish on the printed page. So Office reduced the size, knowing how often we hit the “Print” button. Graphic designers would actually say that between 9 and 11 point font sizes are best for reading evaluation reports. For Web-based dissemination, you can comfortably go a little larger. You wouldn’t want your report to appear comical because of your choices in font.

Hot tip: Speaking of screen readability over print, differences also exist between fonts that can impair readability. Have you ever opened an attachment only to wonder whether the sender had consumed her requisite coffee for the morning because the text is wacky? Fonts don’t always translate across computers. Safe picks for screen reading are Verdana and Georgia. Helvetica and Times Roman are also quite likely to be on most computers and they were designed to be read on the printed page. Pick one or two fonts and keep them consistent across all of your evaluation work with a single client.

Hot tip: Fully justified text looks crisp, professional, and formal, if proper attention can be given to ensure wide gaps don’t appear on certain lines. Left justified text creates a more informal flow, but it is easier to read. Which do you want to communicate to your client? (Hint: Right or center justified text is difficult to read so use it sparingly only for headers or titles of reports.)

Hot tip: Graphic designers tend to disparage bullets as a way to establish emphasis in a written report. Try to highlight your evaluation’s main points with an indent, font change, alignment difference, bold, italic, or even symbols (NOT webdings. Sorry.). Just don’t overdo it. At most, pick two ways of highlighting to combine (i.e., italic indent) and be consistent about it in all of your communications with one client.

Rad Resource: I borrowed these ideas from Ellen Lupton’s (2004) totally awesome, totally accessible book, Thinking With Type. Her website, thinkingwithtype, has visual examples and games to underscore the Hot Tips I listed here, plus lots of others (like how we MUST stop putting two spaces between our sentences).

Want to hear more from Stephanie on Graphic Design for Evaluators? Sign up for her AEA Coffee Break Demonstration Webinar to be held on Thursday, July 22. Learn more on AEA’s CBD page.

Graphic design, reports, type

Hello, I am Melvin Mark, Professor and Head of Psychology at Penn State University. When you read books or articles about evaluation, the focus typically is on doing an upcoming evaluation. Given that conducting individual evaluations is what evaluators are usually hired to do, this focus of our books, articles, and conversations makes sense.

Hot Tip: There are a set of questions that are not about the conduct of an individual evaluation that might deserve more of our attention. Consider a few examples:

• What gets evaluated and why? For instance, do evaluation funders tend to focus on questions for youth and the disadvantaged?
• Collectively, should we try to help to bring about evaluation of certain programs or policies that have escaped evaluation (e.g., should we encourage evaluators in academic settings to take on certain work pro bono)?
• What should our professional associations try to do, beyond offering professional development, standards and principles, conferences and articles that focus on individual evaluations?
• What different roles might evaluators (and others) legitimately take on in efforts to facilitate the use of evaluation?

Exploring such questions can be fun. Moreover, I think it can help us to improve the way we conduct evaluations, to act in ways that are both ethical and useful, and to bring value to individual evaluators, to those we serve, and to the field at large.

Want to explore these questions, and others, with Mel? He will be serving as the discussant for the week of June 20-26 on AEA’s Thought Leaders Forum. Learn more online here: http://www.eval.org/thought_leaders.asp

questions, thought leaders

Greetings colleagues. My moniker is Michael Quinn Patton and I do independent evaluation consulting under the name Utilization-Focused Evaluation, which just happens also to be the title of my main evaluation book, now in its 4^th edition. I am a former AEA president. One of the challenges I’ve faced over the years, as many of us do, is making evaluation user-friendly, especially for non-research clients, stakeholders, and audiences. One approach that has worked well for me is using children’s stories. When people come to a meeting to work with or hear from an external evaluator, they may expect to be bored or spoken down to or frightened, but they don’t expect to be read a children’s story. It can be a great ice-breaker to set the tone for interaction.

Hot Tip: I first opened an evaluation meeting with a children’s story when facilitating a stakeholder involvement session with parents and staff for an early childhood/family education program evaluation. The trick is finding the right story for the group you’re working with and the issues that will need to be dealt with in the evaluation.

Rad Resource: Dr. Seuss stories are especially effective. The four short stories in Sneeches and Other Stories are brief and loaded with evaluation metaphors. “What was I scared of?” is about facing something alien and strange — like evaluation, or an EVALUATOR. “Too Many Daves” is about what happens when you don’t make distinctions and explains why we need to distinguish different types of evaluation. “Zaks” is about what happens when people get stuck in their own perspective and can’t see other points of view or negotiate differences. “Sneeches” is about hierarchies and status, and can be used to open up discussions of cultural, gender, ethic, and other stakeholder differences. I use it to tell the story, metaphorically, of the history of the qualitative-quantitative debate.

Hot Tip: Children’s stories are also great training and classroom materials to open up issues, ground those issues in a larger societal and cultural context, and stimulate creativity. Any children’s fairy tale has evaluation messages and implications.

Rad Resource: In the AEA eLibrary I’ve posted a poetic parody entitled “The Snow White Evaluation,” that opens a book I did years ago (1982) entitled Practical Evaluation (Sage, pp. 11-13.) Download it here http://ow.ly/1BgHk.

Hot Tip: What we do as evaluators can be hard to explain. International evaluator Roger Mirada has written a children’s book in which a father and his daughter interact around what an evaluator does. Eva is distressed because she has trouble on career day at school describing what her dad, an evaluator, does. It’s beautifully illustrated and creatively written. I now give a copy to all my clients and it opens up wonderful and fun dialogue about what evaluation is and what evaluators do.

Rad Resource: Eva the Evaluator by Roger Miranda. http://evatheevaluator.com/

communication, Eva, metaphors, stories

My name is Mike Morris and I’m Professor of Psychology at the University of New Haven, where I direct the Master’s Program in Community Psychology. My research focuses on ethical issues in evaluation, and I am an Associate Editor of the American Journal of Evaluation. The best book I’ve ever read for managing my relationships with stakeholders in an evaluation was not written by an evaluator, nor was it written specifically for evaluators.

Rad Resource: Peter Block (2000). Flawless Consulting: A Guide to Getting Your Expertise Used (2^nd ed.). San Francisco: Jossey-Bass. http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0787948039.html

Among organizational consultants this book is legendary. Evaluation is, in my view, one form of consultation, so it’s not surprising that Block’s book is relevant to our work. His discussion of such issues as entry/contracting, dealing with resistance, and managing the feedback of results is invaluable. Central to his analysis is the concept of “authenticity,” which means putting into words what you are experiencing with stakeholders as you work with them. It might sound a bit scary at first, but the more you practice it, the more effective at managing these relationships you become. I also believe that Block’s approach to consulting can enhance the ethical quality of evaluations, especially in terms of helping evaluators identify strategies for raising and pursuing ethical issues with stakeholders.

Flawless Consulting is exceedingly well-written. It probably helps that Block does not have a doctoral degree, since writing a dissertation is a process that can extinguish one’s ability to compose a sentence that anyone would be interested in reading. Flawless Consulting gets very positive reviews from my students. I hope you’ll agree with them.

Want to learn more from and with Mike? He will be the guest for the week of April 18-24 on the AEA Thought Leaders Online Discussion Forum. Learn more at http://www.eval.org/thought_leaders.asp

consultants, project management, stakeholders

My name is Amy A. Germuth, President of EvalWorks, LLC (http://EvalWorks.com) and owner/blogger at EvalThoughts.com. I’ve worked over the last year on improving my evaluation reports to better meet my client’s needs and have a few great resources to help you do the same.

Rad Resource: “Unlearning Some of our Social Scientist Habits” by Jane Davidson (independent consultant and evaluator extraordinaire, as well as AEA member and TIG leader). http://davidsonconsulting.co.nz/index_files/pubs.htm She recently added some additional thoughts to this work and presented them at AEA’s 2009 annual conference in Orlando. Her PowerPoint slides for this presentation can be found at: http://bit.ly/7RcDso.

Frankly, I think this great article has been overlooked for its valuable contributions. Among other great advice for evaluators (including models or theories but not using them evaluatively and leaping to measurement too quickly), she addresses these common pitfalls when reporting evaluation findings: (1) Not answering (and in some cases not even identifying!) the evaluation questions that guided the methodology, (2) reporting results separately by data type or source, and (3) ordering evaluation report sections like a Master’s thesis. This entertaining article and additional PowerPoint slides really make a case for using the questions that guide the evaluation to guide the report as well.

Rad Resource: The “Evaluation Report Checklist” by Gary Miron (professor at Western Michigan University and former Chief of Staff at The Evaluation Center at WMU) provides a great outline of the eight main sections in an evaluation report (Title page, Exec. Summary, Table of Contents, Introduction and Background, Methodology, Results, Summary and Conclusion, References) and the various things that should be included in each. http://www.wmich.edu/evalctr/checklists/checklistmenu.htm

The author notes that this checklist can be used as a “tool to guide a discussion between evaluators and their clients regarding the preferred contents of evaluation reports and a tool to provide formative feedback to report writers” and can help writers identify the strengths and weaknesses of their report. However, as Gary  notes, evaluation reports differ greatly in terms of purpose, budget, expectations, and needs of the client, thus one may need to consider or weight the checkpoints within sections as well as the relative importance and value of each section when reviewing one’s own writing (or someone else’s).

Using the Evaluation Report Checklist in conjunction with some of Dr. Davidson’s suggestions has increased the quality and utility of my evaluation reports and should do the same for yours.

This contribution is from the aea365 Daily Tips blog, by and for evaluations, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org

checklist, communication, reports