My name is Chris Lysy, I am a Research Analyst at Westat, the creator of Eval Central, and cartoonist at freshspectrum.  You can also find me on twitter (@clysy).

At first cartooning was just a hobby, but over time I’ve started to discover the utility of a well designed cartoon.  The right cartoons can be potent tools for introducing complex concepts, engaging audiences, and sparking discussion.

On my blog I’ve been experimenting with the fine line between chart junk and good illustration design.  A topic briefly addressed by Edward Tufte on his website.  Here are some tips I came up with to help you avoid crossing the line.

Hot Tip: 95% Concept, 5% Illustration. If you have a great concept, the illustration doesn’t have to be that great.  I spend the vast majority of my time on the concept.  Check out the following cartoon from my attribution post.  Not the best illustration, but the point is made.

Hot Tip: Be Abrupt. Subtlety thrives in other forms of communication.  You have a few seconds to capture attention, just say what you want to say.  Here’s an example from my What is Evaluation? post.

Hot Tip: Costello your Abbott. Visuals should always complement the point, not simply repeat the point.  In the famous sketch, Costello didn’t completely parrot Abbott, and your visuals should not completely parrot your points.  Here is an example from my counterfactual series.  My cartoon complements David Henderson’s point.  They work together.

“Outliers do not make for compelling client testimonials. Use your metrics to identify what the average experience in your program looks like, and get testimonials from people who fit this profile.”

Hot Tip: Push the Limits. Cartoons give you a certain latitude to go over the top.  When used wisely, pushing the limits can spark great discussion.  Here is one of my early cartoons from a post on how to not really evaluate.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

cartoons · data visualization

Our names are Marc Wheeler and Salem Valentino and we are internal evaluators for Big Brothers Big Sisters of America.

There is a lot of buzz today about infographics.  Many of you may have thought about using infographics in your evaluation reports to try to translate your findings more effectively.  We recently took the plunge and incorporated infographics in our 2013 Youth Outcomes Report .

For example, we created the infographic below to clarify the theoretical connection between the short-term outcomes we currently measure for each youth and those long-term outcomes of interest to many stakeholders. The graphic summarized a large quantity of research literature in a single infographic that was easily interpretable and concise.

Lessons Learned:

1. High-quality infographics require a certain level of expertise; as we didn’t have the relevant experience in house, we contracted with an external graphic designer who delivered great results.
2. Beyond experience, it also takes time to develop your infographics and get them right.  First, you need to figure out the story behind your data.  Then, what are the best ways to illustrate this story, while remaining true to the data?  For instance, we wanted to communicate effect sizes but didn’t want to take up space in our report explaining what they meant to a lay audience.  So instead, we developed the visualization below to better illustrate our story of the magnitude of youth outcomes.  Lastly, infographics require a number of iterations and can benefit from the input of diverse audiences.  Budget your time accordingly.

1. Understanding your project needs will help you choose the right designer. We looked at several resources on the internet to find a graphic designer. For us it was important that our work with the designer was collaborative; we wanted to ensure the quality of the evaluation content. Due to our timeline, it was also important that our designer could design the entire report and not merely the infographics.  You may also want to ask your designer how comfortable they are with Excel or other types of data you will be using in the report.

Rad Resources:

Elissa Schloesser at Visual Voice – our designer’s 5 Steps for Translating Evaluation Findings into Infographics

Visual.ly’s Marketplace service will find a designer for you and help you create an infographic for one price.

Easel.ly is a website where you can create your own infographic for free.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization · evaluation reporting · infographics

Hello, I am Maggie Miller, the principal of Maggie Miller Consulting.  I conduct program evaluation for small to mid-size nonprofits in the Denver/Boulder area.  Sometimes I think I should change that tagline because I’ve gotten referrals to larger nonprofits (whom I adore), but I am still very attached to the small identity.

Hot Tips:

• Trust that being small can be an advantage: when people hire you, they do so because you’re you; you’re not going to be delegating to staff.
• If you’re really small, you won’t have employees, so…
• Have subcontractors instead.  I work with some wonderful people who do data entry, multicultural interviews, and web development.
• Try to get some work as part of a team.  This will give you the camaraderie and learning opportunities that you would have if you worked for a larger company.
  • Know what you’re good at; know what has your name on it.
  • Know your limits:
  • When asked, “Oh, you consult with nonprofits…do you do strategic planning?” the answer is NO.  (Unless you do that too.) Get to know consultants who do related work, and make referrals.
  • Don’t take jobs that are too big.  Be happy to refer people to the larger shops in town.
    • Try to get one gig that brings in 40%-60% of your income, then you can be free to help smaller clients with the rest of your time.

Lesson Learned:

• The better you know what you can and cannot do – and what you like to do and don’t like to do – the better able you will be to attract your perfect clients.
• It’s all about relationships.  Enjoy them and keep them strong.

Rad Resources:

• This book: Attracting Perfect Customers: The Power of Strategic Synchronicity by Stacey Hall and Jan Brogniez.  This is somewhat woo-woo (i.e., law of attraction) but has guided my business philosophy and is full of useful exercises that brought me wonderful clients!
• All of the American Evaluation Association’s (AEA’s) amazing and affordable educational offerings, and fabulous resources like Stephanie Evergreen’s blog and Ann Emery’s Excel videos.
• See other AEA365 posts on Small is Beautiful

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. 

data visualization · Independent Consulting

Hello! I am Manny Straehle and I work with International Credentialing Associates (ICA). ICA is a measurement consulting firm with expertise in developing certificates, certification, and licensing programs. We are committed to producing reports and visualizations that are simple and easy to understand for the clients we serve, but we need your help.

Hot Tip – Enter our Data Visualization Competition: With the sponsorship of AEA’s Data Visualization and Reporting TIG, we are launching a data visualization competition. The winner will receive $500.

ICA often generates data reports for our clients – and revising the look and ease of this data report is the basis for this data visualization competition.

Don’t worry if you aren’t familiar with certification and licensing programs, or if some of the terms below are new to you. The contest involves improved organization and display – no data analysis required.  The submissions will be reviewed by our panel of judges, including me, Rory McCorkle, DVR TIG founder Stephanie Evergreen, DVR TIG Co-chair Johanna Morariu, and other experts.

Task:       To create a data visualization-centered report, using a provided dummy set of data.

Hot Tip – Include These Required Report Components:

• ICA’s name, logo, and color schema (available from the ICA website)
• Space for customer name
• The following data points, from the provided Excel spreadsheet:
  • Key (correct answer)
  • P+ (item difficulty)
    • Percent of candidates who answer the question correctly
  • Distractor Analysis (number and percentage) across three performance levels
  • Rbis (item discrimination)
    • Correlation between performance on an item and total performance
  • Recommended actions

Hot Tip – Attend to the Evaluation Criteria:

• Uses basic spreadsheet software (Excel, Open Office, etc.) for data visualizations
• Report fits on one page
• Required report components are present
• Text is legible
• Data visualizations are easily interpreted
• Transfer of graphs from spreadsheet to report is easy
• Report uses best practices in data visualization design
  • Layout and use of space
  • Use of color and contrast
  • Inclusion of appropriate text
  • Use of data visualizations appropriate to data
  • Contribution of design to overall comprehension of information

Hot Tip – Follow Our Submission Instructions:

• Download the Excel spreadsheet with a data set you can use (please note the file is dummy data).
• Submit your revision, both report in PDF format and spreadsheet file, to info@intlcred.com by 15 April 2013.
• The winner will be announced on 15 May 2013 and contacted by email.
• All entries become the property of ICA upon submission.  By submitting an entry, all Entrants agree to the Official Rules of Entry.
• Questions can be directed to info@intlcred.com

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization · reports

Hello all, I’m Kim Firth Leonard, American Evaluation Association (AEA) member since 2008, and President of local affiliate OPEN, the Oregon Program Evaluators Network. I currently work at Marylhurst University in Portland, Oregon, primarily on assessment of student learning and academic programs, though I also dabble in institutional research. I also do contract work in program evaluation via Leonard Research and Evaluation LLC.

Rad Resource – actionable data: I started the blog actionable data in 2011 and post somewhat regularly (a few times per month whenever possible) with a handful of friends and co-authors. The blog advocates for the collection of meaningful and useful data, and for wise use of that data. Our posts span a range of topics often related to program evaluation, though most focus more specifically on data and data use.

Hot Tips – favorite posts: Here are a few mostly recent, favorite and/or most visited posts authored by yours truly so far:

• 11/29/2011 – What data is not actionable? An attempt to further define actionable data is a post in which I babble about what I think actionable data is (and is not) and pose several questions that I myself don’t have answers to.
• 11/23/2012 – I posted on My love/hate relationship with pie charts. Though my opinion is already shifting on this topic, I love the conversation this post generated.
• 12/31/2012 – The Functional Art and growing as an actionable data advocate is the first in what I hope will be a series of posts about my experience in Alberto Cairo’s Infographics and Data Visualization MOOC (massive open online course) and includes a New Year’s resolution declaration.
• 1/11/2013 – Most recently, I posted Analysis Paralysis discussing decision fatigue, noting that “if we want decision makers to use data, we should probably understand decision fatigue.”

(Share Clip)

Lessons Learned – why I blog: For me, blogging is an opportunity to question, explore, and learn as well as to share what I know. To think together with my co-authors and anyone willing to read (and comment) along with us! A ‘manifesto’ for actionable data is here.

I also ‘micro blog’ on Twitter (@KimFLeonard), which has been a wonderful way to engage others with my blog and to find people who are doing interesting work. Between the blog and Twitter, I have discovered many wonderful resources and connected to other great evaluators (including Sheila B. Robinson, who is graciously co-authoring a series of posts with me).

Lessons Learned – what I’ve learned: How liberating and enlightening it can be to throw an idea online. Or to ponder something ‘out loud.’ And that blogging, especially when accompanied by conversation via social media, can be an amazing networking and learning tool.

This winter, we’re continuing our series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Data · data visualization · twitter

I’m Kathleen Tinworth and I co-chair the recently re-named Arts, Culture, and Audiences TIG of AEA with Don Glass, who began this week’s AEA365 series. I lead the Audience Insights department at the Denver Museum of Nature & Science and also consult via my alter ego, ExposeYourMuseum.

Lessons Learned

• Don started this week with a truism about evaluation in arts and cultural settings: “outcomes and outputs…are sometimes inventive, innovative, and unpredictable.”
• Jessica Sickler provided a great anecdote of exactly that, writing about interviewing while a child tied a stuffed snake around her legs!
• The work lends itself to creative tools, instruments, and measures—for example, the timing and tracking method outlined in Amy Grack-Nelson’s post.
•  That said, there are often real challenges associated with defining audience outcomes, gathering data in ever-moving, highly social environments, and promoting the value of evaluation to arts and culture organizations and stakeholders, as Joe Heimlich underscored.
• “Performing arts organizations,” Jennifer Novak-Leonard reminded us “are in the business of transforming individuals through arts experiences, but evaluation is rarely on their radars and box office receipts and the number of ‘butts in seats’ are used as proxies of how their art impacts and transforms individual people.”

To combat the challenges above you might assume that arts, culture, and audience evaluators have mastered creativity and innovation when it comes to reporting, presenting, and dissemination– ensuring our communication is as vivid and inspiring as the environments in which we work. Here’s a secret: we haven’t. (Just asked Stephanie Evergreen, who critiqued more museum evaluations than any person should ever have to for her PhD dissertation.) As an evaluator in this sector, and as an AEA TIG co-chair and board member of the Visitor Studies Association, prioritizing good, clean, accessible evaluation communication tops my “OMG that’s gotta change NOW” list.

Rad Resources

• The emergence of the Data Visualization and Reporting TIG and AEA’s Potent Presentations Initiative have been two AEA-specific resources that help me push my work to more effectively convey valuable insights. There were great sessions at AEA2012 in Minneapolis this year, rethinking evaluation reports and reimagining how we communicate.
• Check out one of my favorites: Kelci Price (Colorado Health Foundation) channeled Nancy Duarte, challenging evaluators to ask ourselves “How badly do I want my idea to live?” Her session, “Crafting Powerful Reports and Presentations: Strategies for Improving Communication in Evaluation” is online in AEA’s public library.

Thanks for joining us this week and come visit ACA sometime soon.

The American Evaluation Association is celebrating Arts, Culture, and Audiences (ACA) TIG Week. The contributions all week come from ACA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

communication · creativity · data visualization · dissemination

My name is Kylie Hutchinson.  I am an independent evaluation consultant and trainer with Community Solutions Planning & Evaluation.  I give regular webinars and workshops on evaluation topics for the AEA and CES and Twitter weekly at @EvaluationMaven.

Like all of you, I try to be conscientious about conducting the best evaluation possible.  But for many years I felt that my final reports were missing the mark in terms of their utilization potential.    Unfortunately, too often evaluation reports end up on the shelves of many a decision-maker collecting dust if they’re not structured and presented properly.

Hot Tip: Attend the AEA eStudy:  An Executive Summary is Not Enough: Effective Reporting Techniques for Evaluators. Reporting is an important skill for evaluators who care about seeing their results disseminated widely and have their recommendations actually implemented. This webinar presents an overview of three key principles for effective reporting and engages participants in a discussion of its role in effective evaluation.  Attendees will leave this webinar with an expanded repertoire of innovative reporting techniques.

Hot Tip: Attend a Training: The interest in, and attention to, more effective reporting has increased exponentially among evaluators over the past few years.  Several other initiatives the AEA has undertaken over the past few years include forming a Data Visualization and Reporting Topical Interest Group (DVRTIG) and the new Potent presentations initiative. Sign up for a 30 minute training on July 17 from 11-11:30 or July 26 from 4-4:30 EST.

 Hot Tip:  Attend the Ignite Sessions at the AEA Annual Conference. The best ticket at last year’s AEA conference was the Data Visualization TIG’s Ignite session and business meeting, so arrive early this year for a good seat.

Rad Resources:

• Gary Miron’s Evaluation Report Checklist
• Jane Davison’s Editorial: Unlearning Some of our Social Scientist Habits
• 2012 Summer Institute Handout – An Executive Summary is not Enough: Effective Evaluation Reporting Techniques

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

checklist · data visualization · eStudy · Potent Presentations · twitter

Hey there. I’m Stephanie Evergreen, AEA’s eLearning Initiatives Director and general data communications geek. Susan Kistler has a family obligation this weekend, so I’m stepping in to share with you the newest developments in AEA’s Potent Presentations Initiative (p2i).

You’ve heard about p2i, right? It is a new initiative to help AEA members improve their presentation skills, particularly around delivering conference presentations. We come together once or twice a year to teach each other about our practices and processes, so shouldn’t we do everything we can to make it easy to learn from our presentations? That’s why p2i will feature online and in-person training before and during the annual conference around the three facets of presenting: message, design, and delivery.

We have just launched p2i.eval.org, which will be the hub of this activity.

Rad Resource: Our home page features our upcoming webinar-based training on how to prepare for and deliver an Ignite session. When you receive the proposal status notice for your Ignite session on July 3, head to our site to sign up for one of the two trainings, either on July 17 at 11:30am ET or July 26 at 4pm ET.

Rad Resource: Our first tool to help you rock your conference session is the Presentation Preparation Checklist. Download this PDF to find out what to prepare when, keep yourself on track, and minimize the last minute rush many people experience leading up to a conference presentation. The checklist include time frames specific to this year’s annual conference, October 22-28.

Rad Resource: During the conference we’ll provide a demonstration on research-based effective practices around slide design. But you don’t want to wait until then to begin working on your session slides. So we’ve released the handout for that demonstration already. Head to the p2i site to snag the Slide Design Guidelines (with extra tips for handouts, too). It covers how to handle fonts, graphics, colors, and arrangement and includes links for step-by-step instructions (we’ll add links each month) and awesome extensions of these guidelines from your AEA colleagues.

annual conference · conference · data visualization · p2i · presentations · training

Hi, I’m Sara Plachta Elliott, Evaluation Fellow at the Skillman Foundation in Detroit, Michigan, through a grant to Brandeis University’s Center for Youth and Communities. As an Evaluation Fellow, I work with Foundation staff to create learning opportunities that inform the work of improving school and neighborhood conditions for Detroit’s kids.In 2010, Brandeis conducted a System of Supports and Opportunities (SOSO) analysis in six neighborhoods where Skillman focuses its grantmaking and changemaking efforts. Data collection involved interviewing youth program and basic service provider staff to assess program offerings, participation rates, program quality, and accessibility.

Below are learning opportunities that were created with SOSO data.

Lesson Learned: Have stakeholders review findings before the final report is prepared.

• An internal review with Foundation staff helped identify inaccuracies in participation rates and site location addresses.
• Then two-page SOSO snapshots for each neighborhood were released.
• Key neighborhood and youth program stakeholders were engaged in a review of findings and lifted up questions about how data were collected. This review process ensured “on the ground” validity of the analysis.
• For full transparency, the dataset was also released to partners for system planning purposes.

Hot Tip: Create maps or other visual products.

The Foundation engaged its partner Data Driven Detroit to create maps of agency and program site locations. In meetings with stakeholders, we reviewed these maps along with youth population maps. In one neighborhood, we learned that most kids lived on the west side but program sites were clustered on the east side. The collective “ah ha” moments helped the Foundation and its stakeholders work together to fill gaps.

Consider hosting a discussion session and ask community partners and stakeholders to interpret brief data reports and maps. What patterns do they see in the data?

Cool Trick: Engage youth and residents in reviewing data, not just program staff.

In the summer of 2011, a social service agency, Southwest Solutions, organized a community youth mapping project. Youth reviewed the Brandeis SOSO data, then designed and conducted their own neighborhood opportunities survey.

Youth researchers were paid through summer youth employment funding and walked every street in their neighborhood twice, administering surveys to youth, youth program staff, and business owners, as well as mapping vacant properties. They discovered that some local businesses wanted youth to work during school hours, thereby tempting them to drop out of school. The youth also learned that students wanted more college and career preparation opportunities, mirroring findings from the SOSO report that more of these opportunities were needed.

Evaluation reports can sit on a shelf if not accompanied by intentional learning opportunities. As an evaluator, encourage clients to create interactive learning opportunities with short, visual reporting products. Spark their collective learning!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

census · data visualization · Evaluation · grantmaking · neighborhood mapping · stipends · System of Support · youth focused evaluation · youth participatory evaluation · youth researchers

My name is Julie Lo and I’m a research assistant at Public Profit.  We help youth service agencies and educational organizations manage what matters.  I want to share with you a way of presenting survey data through an embedded interactive interface created using the program Tableau Desktop and later shared online using Tableau Public.

We process thousands of stakeholder surveys each year for our clients and were thrilled when one of them approached us about creating a method of publishing a city-wide survey of thousands of high school youth to a wider audience.  We found a way to consolidate what in past years has been a 250-page report to an interactive set of data visualizations:

• Interactive Fall 2011 Student Survey Results

Emphasizing the many different things to consider when creating any kind of data visualization, I’ve included some practices that we will be sure to consider for future projects like this.

Lesson Learned: Which came first: function or design? 

The first leg of the project involves building out of the basic frame: pages, buttons, selecting the best viz to tell the story. Leave enough fuel in your project’s gas tank to be able to do this. Share it with your client for feedback to make sure that it not only works well, but is also designed in a way that is accessible for youth, teachers, and policy analysts alike!

We worked through several drafts before arriving at the final product both internally and with our client.  This helped us to make decisions around what types of visualizations work best for the data on hand.  Luckily, our client has been working on this project for multiple years and was able to share valuable insights on what information to emphasize for the audience and the “look” that would be most appealing.

Lesson Learned: How fit is your data?

Tableau Public has excellent built-in online sharing features and is seamlessly linked to Tableau Desktop making the work of transferring your work online quick and easy. However, Tableau Public does not have a way to suppress the underlying data, posing potential issues around the confidentiality of individual survey takers.  We got around this by using summarized data that we prepared outside of Tableau in SPSS and used summarized data for the underlying data for the Tableau Public file.  This is an important step to consider when planning your project timeline since it adds a little bit of data-prep work.

Lesson Learned: So many questions, where to begin? 

We worked closely with our client to help develop themes to group survey items together and to select ways to filter data by demographics. This creates further utility for both our clients and those they serve. Presenting questions grouped together by theme can help viewers navigate survey results for their populations of interest, helps to make sense of data and keeps pages free of the dreaded data-viz clutter.

Rad Resources – Related aea365 posts:

• Nina Potter on Tableau for Data Visualization
•  Susan Kistler on Training and Data Visualization

Rad Resources:

• Show Me the Numbers: Designing Tables and Graphs to Enlighten, Stephen Few
• Freakalytics

We’re celebrating Data Visualization and Reporting Week with our colleagues in the DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

data visualization · Tableau