AEA365 | A Tip-a-Day by and for Evaluators

CAT | Data Visualization and Reporting

Hello! I’m Molly Ryan, a Research Associate at the Institute for Community Health (ICH), a non-profit in Cambridge, MA that specializes in community based participatory research and evaluation. I am part of the team evaluating the Central Massachusetts Child Trauma Center (CMCTC) initiative, which seeks to strengthen and improve access to evidence-based, trauma-informed mental health treatment for children and adolescents. I would like to share a great resource that we use to visualize and communicate findings with our CMCTC partners.

Rad Resource: Icon Array University of Michigan researchers developed Icon Array to simply and effectively communicate risks to patients. For more information on why icons are so rad, check out Icon Array’s explanation and bibliography.

Hot Tip: Icon Array offers 5 different icons to choose from.

6 out of 11 reassessments (54.5%) received

6 out of 11 reassessments (54.5%) received

Hot Tip: Icons aren’t just for risk communication! We use icons to help our partners understand and visualize their progress collecting reassessment data for clients.

14 out of 24 reassessments (58.3%) received •9 out of 14 (64.3%) complete •5 out of 14 (35.7%) incomplete

14 out of 24 reassessments (58.3%) received
• 9 out of 14 (64.3%) complete
• 5 out of 14 (35.7%) incomplete

Cool Trick: Icon Array allows you to illustrate partial risk by filling only a portion of the icon. We used this feature to communicate whether a reassessment was complete or incomplete for a given client.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Lisa Kohne, an independent evaluator and I work for SmartStart Consulting in Orange County, California. We specialize in conducting project evaluations for federally funded grants, primarily from the National Science Foundation. Most of our clients are math, science, and engineering professors from four-year universities. One of our big challenges is that some of our projects are multi-institutional, multi-state, and multi-country. It’s very difficult to bring multiple partners together at the same time to discuss evaluation findings – and most don’t have the time, inclination, or enough evaluation knowledge to read lengthy reports.

Hot tip:
To overcome these challenges we began to develop Evaluation Newsletters. They are usually two pages with lots of graphs, maps, and tables. We try to make them colorful, high-interest, and eye-catching. Some are wordier than others and our “skills” have evolved over the years. You can see the evolution from one of our earlier version (very wordy) to our more recent version (less wordy, more white space).

Kohne 2 Kohne 1

We only offer these to our larger, multi-site projects.  The reactions and feedback have been extremely positive.  No PI has ever turned down the offer to create a newsletter.  They are also great to distribute at advisory board meetings and project conferences.

Rad Resources:

  • Google Images works great for the simple clipart needed for newsletters. Simple is better. Just be careful to not use copyrighted ones.
  • Microsoft Publisher is our current choice of software.  We’ve tried Word but Publisher lines up the information much better.  Also, the new online subscriptions to MS Office 365 include Publisher.

Kohne 3

SmartArt is our go to graphic developer.  Only available in MS Word, not Publisher.  So you need to create it in Word and paste it into Publisher.

Lessons Learned:

  • Less is more.  Less words, more pictures, lots of bullet points.
  • Make it personal and make it positive.  Add university and project logos, project goals, maps that indicate locations of participating institutions, funders’ logs, and anonymous quotes from participants.
  • Newsletters take a lot of time so build the cost into the budget.
  • Recruit your most artistic employee to create your newsletters – someone who understands color, balance, and brevity of words.
  • Send a sample to stakeholders and ask if it would be helpful to get evaluation results out.
  • Get commitment from your principal investigator to email the newsletters out to all stakeholders and/or project participants and post them on their project webpage.  Here is a webpage containing out newsletters on a NSF PIRE project.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hey friends! We are Ann Emery, Co-Chair of the DVRTIG, and Stephanie Evergreen, Founder of the DVRTIG – two evaluators who are crazy about data visualization.

“I love your examples, but how do I know what I should do next time I’m creating a graph?” We heard comments like this when we talked with evaluators about good graph design. We saw that evaluators had thirst for better graphs and a clear understanding of why better graphs were necessary, but they lacked efficient guidance on how, exactly, to make a graph better.

Rad Resource: Introducing the Data Visualization Checklist

Take the guesswork out of your next graph. Download our 25-item checklist for clear guidelines on graph text, color, arrangement, lines, and overall messaging. Read about what makes a memorable graph title (spoiler alert: it’s not “Figure 1”). Learn how to arrange your bar chart based on whether your categories are nominal or ordinal. Decide which default settings in your software program to keep and which ones to toss.

Emery 2

Not familiar with the terminology? The last page is a Data Visualization Anatomy Chart. Watch that example’s before-and-after remake in Stephanie’s training on Ignite presentations.

Hot Tip: How can you use the Checklist?

Get in the habit of producing several drafts before sharing final graphs with your clients. Draft, score, edit, repeat!

In this example, we printed an existing graph (page 6 here) in both color and black and white to see how the final chart looked for viewers. We scribbled notes all over the graph and the checklist as we scored. Overall, the graph earned 91% of the possible points—just above the cutoff that enables viewers to read, interpret, and retain the content.

Emery 1

Cool Trick: What’s next for the Checklist?

We are publishing examples to illustrate the 25 items as well as before-and-after remakes. Check out the growing galleries at http://annkemery.com/tag/data-visualization-checklist/ and http://stephanieevergreen.com/tag/data-visualization-checklist/. And we’re taking requests: Which checklist items would you like examples for?

We’re also hoping to present the checklist at the American Evaluation Association’s annual conference in October. Let’s high five there! Please please please can you take a picture of your existing data visualization, apply the checklist, and then take another picture? Tweet your redesigns to @annkemery and @evalu8r. Email them to annkemery@gmail.com and stephanie@evergreenevaluation.com. Fold them into paper airplanes and fly them to us! Send your redesigns and show people how awesome you are!

Big thanks to our pilot reviewers: James Coyle, Amy Germuth, Chris Lysy, Johanna Morariu, Jon Schwabish, David Shellard, Rob Simmon, Kate Tinworth, Jeff Wasbes, and Trina Willard.

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Greetings AEA365! My name is Gretchen Biesecker, and I am the Vice President of Evaluation at City Year. City Year is an education-focused, nonprofit organization founded in 1988 that partners with public schools and teachers to keep students in school and on track to succeed.

This year I competed in my first storytelling slam—an event where people tell five-minute, first-person, true stories. Constructing and telling my story was really fun. I started thinking about new ways our staff at City Year could think about incorporating numbers or data into our communications and reporting.

Lessons Learned:

  • Sharing findings in context is important, especially for audiences that may be unfamiliar with the data. To someone within a school, improving the average daily attendance rate by 2% may be a huge win, but to someone outside education, without context, that increase may sound miniscule.
  • Taking a look at some resources to organize good stories was really helpful to me. Reviewing the ways to construct a good story: 1) Helped me generate ideas for sharing different kinds of numbers and data to share our story; and 2) Emerged as a foundational step before I think about data visualizations and creating reports or presentations.

Rad Resource: Nancy Duarte’s book, Resonate, is now available for free! Duarte shares multiple examples of effective story structures, which may inspire you.

Resonate by Nancy Duarte

Resonate by Nancy Duarte


Hot Tips: Here are some additional ideas you might take from storytelling. You can use numbers to:

  • Create Drama—good stories may be formatted as a sweeping saga, and you can use numbers to convey scale (e.g., City Year serves in 242 schools, reaching 150,000 students). Pairing that with a personal story and results from one child or case is powerful.
  • Set the Stage—numbers can be used to share the problem or give the context for results (e.g., fewer than 40% of students in the nation’s schools score at or above proficiency in English/Language Arts and math).
  • Share the Transformation—good stories have a beginning, middle, and end, or conflict and resolution—so we can share the “before and after” through numbers. You can also “show your work” and the effort or conflict that it took to achieve the results (e.g., in 2012-2013, students in City Year schools spent over 589,100 hours in extended learning time programming).
  • Catch the eye or ear with Repetition—sometimes good stories or speeches have repeating rhymes, words, or numbers, so think about when repeating a particular number may be effective or impactful.

I encourage you to find inspiration and new ideas from the things you love. They may not be within evaluation, but translating them into our field will help us reach more people to put results into action.

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Tony Fujs, Director of Evaluation at the Latin American Youth Center, a DC based non-profit organization. Today, I want to share my experience using R and ggplot2 for data visualization. Ggplot2 is a great tool from the R toolbox (a package, in R lingo). It relies on the powerful Grammar of Graphics framework, which helps “shorten the distance from mind to paper” (Hadley Wickam).

I started using R three years ago, and it has now become my main tool for data analysis and visualization.R is known to have a steep learning curve though, so before getting started, it’s probably a good idea to do a little bit of “cost-benefits” analysis, and check if R is a good fit for you.

Lessons Learned: Benefits

  • More flexibility: R currently meets 99.9% of my dataviz needs. From simple Bar charts, to Maps, to Social Networks… you name it! I can do it directly from R.  No need to learn multiple tools anymore.

Fujs 1

  • Increased productivity:  I often need to generate the same charts on multiple data sets, or multiple subsets of the same dataset. With R it takes almost no effort to do this.

Fujs 2

  • Transparency: Who never did open an Excel file with numbers that seemed to come directly from a magic hat? Coding forces you to become more transparent, and makes your analyses and dataviz easier to replicate. Your colleagues and your future self will thank you for this!

Lessons Learned: Costs

By the way, did I mention that R is free? Your costs-benefits analysis is looking pretty good! You still have some time investment ahead though… In my experience, there are two main barriers when starting dataviz with R and ggplot2:

-       Understanding the Grammar of Graphics: There is a strong logic behind the grammar of Graphics framework. Taking some time to understand it will make your learning experience much smoother.

-       Learning to code: If you don’t have any prior programming experience, moving from a point and click environment to writing code may entail some frustration.

Rad Resources: Here is a list of resources to help minimize learning costs:

1)    Get familiar with the grammar of graphics:

  • Read this paper by ggplot2 creator Hadley Wickam.
  • Check out this slide deck from my dataviz workshop using R and ggplot2. Pay special attention to exercise on slide #25

2)    Get familiar with the R environment:

3)    Start playing with ggplot2

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

We are Rakesh Mohan, Lance McCleve, Tony Grange, Bryon Welch, and Margaret Campbell of the Office of Performance Evaluations, an independent agency of the Idaho State Legislature.

Last year, the Legislature asked us to explain how funds move through the Department of Health and Welfare—the agency with the state’s largest budget. Legislators, including budget committee members, had difficulty understanding the department’s financial information. Given agency complexities and its sheer number of financial transactions (over two million in 2013), no one in recent history had brought together the separate parts of its financial records to explain how the parts function.

Communicating the flow of dollars was the trickiest piece of the study. We considered narratives, tables, and traditional flowcharts. Ultimately, we used Sankey diagrams that helped stakeholders visualize funds moving through the department making them the most useful features of our report.

One diagram shows how dollars flow through a program over a year.

Mohan 1

The second diagram illustrates funds flowing into, out of, and within a program.

Mohan 2

Sankey diagrams depict resources within a system by mapping their flow from an initial set of values to a final set of values. They were especially useful because the lines have widths proportional to their value throughout the system. Legislators could easily see the proportion of each fund throughout the flow. Initially we tried to create the diagrams in Visio, but manually sizing each line was extremely laborious.

Department officials were surprised how clearly our report presented complex aspects of their fund management. They are now using our report for in-house training. They also had us train them on creating Sankey diagrams for future reporting. The feedback from three key stakeholders further illustrates the usefulness of Sankey:

 

“This…really, really was a fine body of work.”

Representative Maxine Bell, Cochair, Joint Finance-Appropriations Committee

 

“The study does much to further enhance the transparency and understandability of the largest budget in state government.”

State Controller Brandon Woolf

 

“[The report] will provide valuable assistance for future planning.”

Governor Butch Otter

Hot Tips:

  1. We used 11×17 paper so the diagram would be big enough to show detail and keep the same orientation as the text.
  2. Subtle colors within similar families were more effective than contrasting colors. We also made colors transparent to clarify intersecting lines.
  3. We wanted to edit text during publishing, so we imported the diagram as an image and added text within the publishing software.

Rad Resources:

  1. We purchased e!sankey, which comes in two versions. The pro version allowed us to link to data in Excel.
  2. Google Charts Sankey
  3. Sankey Diagram History

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Welcome! I’m Ann Emery, Co-Chair of the Data Visualization and Reporting Topical Interest Group (“DVR TIG”). It seems only fitting to tell you more about the DVR TIG through, well, data visualization and reporting! Some highlights:

  • AEA has 48 topical interest groups. They specialize in everything from quantitative methods to advocacy evaluation to data visualization.
  • The Nonprofits and Foundations TIG is the largest, with 1,290 members in 2014. The Lesbian, Gay, Bisexual, and Transgender Issues TIG is the smallest, with 115 members. The DVR TIG falls in the middle, with 821 members this year.
  • 3 out of 4 DVR TIG members are women.
  • Most of the DVR TIG members–92%–have a master’s degree or higher.
  • 28 different countries are represented in the DVR TIG. Roughly 9 out of 10 members are from the United States.
  • The DVR TIG members work in a variety of settings: in nonprofits (33%), private businesses (26%), colleges and universities (26%), federal, state, and local agencies (12%), and school systems (4%).

What about demographic patterns over time? And how does the DVR TIG compare to other TIGs, or to AEA as a whole? This analysis is just the beginning. If you dig deeper into the data, let me know! I look forward to seeing what you find. Have you displayed evaluation findings through a dataviz video like this? How’d it go? I storyboard portions of my workshops and webinars to grab the audience’s attention (especially at the beginning and end of a presentation) and to offer step-by-step explanations of complex charts and diagrams. Do you have questions about how I designed this video? Want to make your own? Share your questions and comments below, or connect with me through @annkemery or annkemery.com. The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

No tags

Hey there! I’m Johanna Morariu, a Director of Innovation Network and the Co-Chair of the DVRTIG. DVR is the Data Visualization and Reporting TIG, and we work within the AEA community and through our evaluation work to improve the quality of communications through better data visualization and improved approaches to reporting evaluation findings.

Morariu 1

You could say the first DVRTIG meeting was in 2010, when Stephanie Evergreen convened a small, rowdy group of us to discuss interest in founding the a new TIG to advance issues related to data and information design and reporting within the evaluation community. Since then, we’ve been a fast growing TIG with more than 800 members. (Thanks Stephanie!)

Since those earliest days, TIG members and leaders have worked hard to develop a knowledge and resource base for members and the broader AEA community. One of those newest resources that we’d like to share is a DVRTIG logic model. Yes, our very own logic model!

Morariu 2

Rad Resource: The DVRTIG Logic Model was developed by DVRTIG leadership coming out of the Evaluation 2013 DVRTIG business meeting. The logic model reflects the ideas and ambitions suggested by TIG members at the business meeting. There are more activities than we can hope to achieve, but this is our record of ideas and possibilities for the TIG for the next two years. Check it out and leave a comment in the AEA resource library.

Rad Resource: Another great resource if you’re interested in learning more about the DVRTIG is our website. Anne Worthington is our TIG webmaster and maintains an excellent resources hub of dataviz resources, DVRTIG videos and content, and much more!

Hot Tip: Did you know there is a forum for chatting directly with AEA members interested in data visualization and reporting? Through the DVRTIG website, you can access the DVRTIG eGroup to start a conversation, get feedback on your visualization or report, or share new resources!

We’ve got a great week of posts lined up for you! First, Ann Emery (my colleague at Innovation Network and DVRTIG Co-Chair) will wow you with some stats and visualizations about the DVRTIG. Up next will be Rakesh Mohan and his colleagues from Idaho State’s Office of Performance Evaluations to explain Sankey diagrams. Then we’ll hear from Tony Fujs of the Latin American Youth Center about using R and ggplot2 for data visualization. From there we’ll turn it over to Gretchen Biesecker, Vice President of Evaluation at City Year, to explore storytelling as an effective communication method. And we’ll wrap up the DVR week with Ann Emery and Stephanie Evergreen with a test drive of their Data Visualization Checklist.

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Susan Kistler, contributing editor at TheSmarterOne.com (we finally launched this month – yay!) and former Executive Director for the American Evaluation Association. aea365 contributors have written before about producing online reports (for example here and here), wherein we learned, among other things, that evaluators are still feeling their way in terms of online reporting. The cost of entry (monetary, time, learning curve, etc.), and the dearth of examples, can make the possibility of reporting online seem daunting. Today’s tool can help.

Rad Resource – ThingLink: ThingLink is an image interaction tool that allows you to add small icons with rich media tags to images which may be viewed on their website or embedded in your own website or blog. I wrote a complete review of ThingLink here and I encourage you to take a look to understand the platform more fully.

Lessons Learned: Using ThingLink it takes from 3-10 minutes to add a set of rich media tags to an image, assuming that you have the image and the content for the tags identified.


STOP HERE! If you are reading this via email PLEASE click back to the aea365 site (just click on the post title in your email). ThingLink is worth seeing in action, but it is a web-based application and works only online. In email, you likely can see the images below, but you won’t be able to interact with them.


Lessons Learned – ThingLink for Graphs and Charts: ThingLink can be used to annotate graphs, such as the one below that I made from data from a US Department of Education report. In this example, the “picture” being annotated is an image of a graph. The ThingLink rollover annotations that I added provide detail to the graph’s categories, scale options, and sourcing information.

Hot Tip: The positioning of ThingLink rollover icons changes slightly among different browsers and platforms. Try to put icons with open space around them and then check how they look.

Lessons Learned – ThingLink for Qualitative Data Gathering and Reporting: ThingLink can be used to annotate photographs. The settings may be configured so that anyone can edit a photo – evaluators could share an image and ask stakeholders to provide written or oral annotations. Alternatively, evaluators may create an annotated photo as part of report, integrating the authentic voice of stakeholders through oral or written annotations. The ThingLink below is annotated with narrative (and a yelp review – wanted to demo that as well!).  

Hot Tip – Try it! As long as you are on the aea365 website (but not in your email), you can add a tag to the image of the bus below. You don’t need to log in. Just click on the pencil icon in the top left of the picture and select edit. Once in edit mode, click anywhere on the picture to add an icon and rollover annotation.

No tags

I’m Susan Kistler, of TheSmarterOne.com, and I’m on to Part III of an Innovative Reporting Series (see Part I on chocolate reports and Part II on adding video to your toolbox). Today I wanted to share lessons from a project marrying street art, evaluation, infographics, technology, and community building.

howtodayLessons Learned: Last month, I had the privilege of interviewing Lisa Koeman, co-lead for the Visualising Mill Road project. The project placed small voting devices in 18 shops on Mill Road in Cambridge, United Kingdom. The devices asked straightforward questions about perceptions of happiness, safety, and social connectedness and had three buttons with which visitors could indicate a positive, neutral, or negative response. The questions were replaced every two days over a three week period and the information gathered was reflected in growing chalked Infographics on the street in front of each shop, with a new line added in the early hours every other day to reflect question responses in that shop.

image004As the project unfolded, they evaluated “the effect of the public visualisation of community data…How can it inform people on what other members of the community think of specific local issues? Does it encourage reflection and discussion?”Although the analysis of the results is still in progress, Lisa was pleased to share that the project encouraged community discussion about the issues raised generally and the project specifically at both the point of data collection (what’s up with those little boxes and buttons?) and reporting (what’s up with those infographics on the sidewalk?). I’ll be sure that we share the final report when it comes out.

image005Hot Tip: For me, this statement from Lisa was the most compelling in terms of rethinking data visualization and reporting: “Lots of people go for screens, for sharing results digitally, but by positioning graphics of local data on the pavement, at people’s feet, it attracted attention and made it accessible for the community. It gave the data back to the people.” She noted that people who would be unlikely to view an online report were talking with others about the graphics and voting devices that had appeared right on their doorsteps.

Rad Resource: Learn more about the project at http://visualisingmillroad.com/.

Hot Tip: Spray painted chalk lasted all three weeks, even through rain, yet faded away over time, leaving no permanent marks.

Hot Tip: Work with community groups to gain buy-in and get permissions and to ensure that the questions asked are meaningful.

Hot Tip: By working with local artists, as part of their 5:00 AM chalking team, they found an affordable way to complete the project and gained further buy-in from an additional group living in the Mill Road area.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top