AEA365 | A Tip-a-Day by and for Evaluators

CAT | Data Visualization and Reporting

Greetings, fellow evaluation enthusiasts! My name is John Murphy and I am an Evaluation Associate on the Education Research and Measurement team at Cincinnati Children’s Hospital Medical Center. We provide evaluative support to various forms of learning and career development, ranging from clinical orientation to leadership and management training. One of our clients does extensive work surrounding quality improvement education. While we have provided them with evaluations of their courses, we have been fortunate enough to glean important tips and advice from their wealth of knowledge in measurement theory. One of the most basic tools used in improvement science is the annotated run chart, a simple form of the line chart. What makes the annotated run chart different from the typical line chart are annotations, small text snippets that show when an intervention or event has taken place.

Murphy

More and more, I have embraced annotations as being crucial to providing context and a story to a data representation. As an aficionado of data visualization, I have begun the quest for the perfect annotation. Here is what I have found so far:

Rad Resource: Stephanie Evergreen and Ann Emery provided an amazing resource within the virtual pages of this very blog! Their data visualization checklist not only reaffirmed my enthusiasm for the annotation, it also gave concrete guides for font size and text direction.

Rad Resource: What discussion about data visualization would be complete without mentioning Edward Tufte? The first chapter of his 2006 book Beautiful Evidence, entitled “Mapped Pictures”, discusses, in rich detail, the benefits of various techniques of providing context to images. Placing content in its proper space, scale, and time is crucial for making all genres of data representation tell a compelling story.

Hot Tip: If you are creating many annotated run charts that are updated frequently, consider investing in BI software such as Tableau. While Excel data labels are functional for one-shot data representations, more dynamic software can save time and provide more flexibility so annotations fit the story instead of being limited by the medium.

Good luck in telling your data stories!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Christine Frank and I am an independent Canadian evaluator. I have a couple of questions for you. Do your reports intrigue your audience or send them for coffee? Do people grasp your message easily?

Although I am best known as a program evaluator, I have also taught courses on business communications and co-authored a textbook on that subject. Experts in business communications focus on dynamic, readable writing. Plain writing experts promote a similar style. Both areas of expertise afford simple strategies to make functional documents more inviting and compelling.

Evaluators sometimes hinder their effectiveness by writing in an overly academic style. For instance, in journal articles, you often find sentences 60 words in length or more. One of the pivotal rules of both business writing and plain writing is to limit sentence length. Even if readers have excellent reading skills and are grounded in the subject matter, you can construct your text to propel them forward, not slow them down. My own frustration in reading unnecessarily lengthy, wordy text drives me to strive for instant clarity.

Hot Tip: For evaluators, I suggest a maximum of 20 words per sentence. You might stretch this limit when a short sentence just won’t convey the message. However, another fundamental rule is to check your text to see if you have used the least number of words possible. If you do this, you may find you can achieve the limit. Many strategies can be applied to maximize clarity. One is to avoid an over-abundance of nouns, especially in sequence. In the following sentence adapted from an actual Request for Proposals, you will see eight nouns, five of them in a row.

  • Our first task is the development of a best practice guideline implementation evaluation plan.

Better

  • First we will develop a plan for evaluating the implementation of best practice guidelines.

Hot Tip: A strategy that reduces sentence length and makes the text more compelling is use of the active voice of the verb.

  • The top three reasons given by students for choosing a career were successfully predicted by teachers.

Better

  • Teachers successfully predicted students’ top three reasons for choosing a career.

Rad Resource: Federal Plain Language Guidelines (2011)

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Christine? She’ll be presenting as part of the Evaluation 2014 Conference Program, October 15-18 in Denver, Colorado.

·

Hello! I’m Molly Ryan, a Research Associate at the Institute for Community Health (ICH), a non-profit in Cambridge, MA that specializes in community based participatory research and evaluation. I am part of the team evaluating the Central Massachusetts Child Trauma Center (CMCTC) initiative, which seeks to strengthen and improve access to evidence-based, trauma-informed mental health treatment for children and adolescents. I would like to share a great resource that we use to visualize and communicate findings with our CMCTC partners.

Rad Resource: Icon Array University of Michigan researchers developed Icon Array to simply and effectively communicate risks to patients. For more information on why icons are so rad, check out Icon Array’s explanation and bibliography.

Hot Tip: Icon Array offers 5 different icons to choose from.

6 out of 11 reassessments (54.5%) received

6 out of 11 reassessments (54.5%) received

Hot Tip: Icons aren’t just for risk communication! We use icons to help our partners understand and visualize their progress collecting reassessment data for clients.

14 out of 24 reassessments (58.3%) received •9 out of 14 (64.3%) complete •5 out of 14 (35.7%) incomplete

14 out of 24 reassessments (58.3%) received
• 9 out of 14 (64.3%) complete
• 5 out of 14 (35.7%) incomplete

Cool Trick: Icon Array allows you to illustrate partial risk by filling only a portion of the icon. We used this feature to communicate whether a reassessment was complete or incomplete for a given client.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Lisa Kohne, an independent evaluator and I work for SmartStart Consulting in Orange County, California. We specialize in conducting project evaluations for federally funded grants, primarily from the National Science Foundation. Most of our clients are math, science, and engineering professors from four-year universities. One of our big challenges is that some of our projects are multi-institutional, multi-state, and multi-country. It’s very difficult to bring multiple partners together at the same time to discuss evaluation findings – and most don’t have the time, inclination, or enough evaluation knowledge to read lengthy reports.

Hot tip:
To overcome these challenges we began to develop Evaluation Newsletters. They are usually two pages with lots of graphs, maps, and tables. We try to make them colorful, high-interest, and eye-catching. Some are wordier than others and our “skills” have evolved over the years. You can see the evolution from one of our earlier version (very wordy) to our more recent version (less wordy, more white space).

Kohne 2 Kohne 1

We only offer these to our larger, multi-site projects.  The reactions and feedback have been extremely positive.  No PI has ever turned down the offer to create a newsletter.  They are also great to distribute at advisory board meetings and project conferences.

Rad Resources:

  • Google Images works great for the simple clipart needed for newsletters. Simple is better. Just be careful to not use copyrighted ones.
  • Microsoft Publisher is our current choice of software.  We’ve tried Word but Publisher lines up the information much better.  Also, the new online subscriptions to MS Office 365 include Publisher.

Kohne 3

SmartArt is our go to graphic developer.  Only available in MS Word, not Publisher.  So you need to create it in Word and paste it into Publisher.

Lessons Learned:

  • Less is more.  Less words, more pictures, lots of bullet points.
  • Make it personal and make it positive.  Add university and project logos, project goals, maps that indicate locations of participating institutions, funders’ logs, and anonymous quotes from participants.
  • Newsletters take a lot of time so build the cost into the budget.
  • Recruit your most artistic employee to create your newsletters – someone who understands color, balance, and brevity of words.
  • Send a sample to stakeholders and ask if it would be helpful to get evaluation results out.
  • Get commitment from your principal investigator to email the newsletters out to all stakeholders and/or project participants and post them on their project webpage.  Here is a webpage containing out newsletters on a NSF PIRE project.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hey friends! We are Ann Emery, Co-Chair of the DVRTIG, and Stephanie Evergreen, Founder of the DVRTIG – two evaluators who are crazy about data visualization.

“I love your examples, but how do I know what I should do next time I’m creating a graph?” We heard comments like this when we talked with evaluators about good graph design. We saw that evaluators had thirst for better graphs and a clear understanding of why better graphs were necessary, but they lacked efficient guidance on how, exactly, to make a graph better.

Rad Resource: Introducing the Data Visualization Checklist

Take the guesswork out of your next graph. Download our 25-item checklist for clear guidelines on graph text, color, arrangement, lines, and overall messaging. Read about what makes a memorable graph title (spoiler alert: it’s not “Figure 1”). Learn how to arrange your bar chart based on whether your categories are nominal or ordinal. Decide which default settings in your software program to keep and which ones to toss.

Emery 2

Not familiar with the terminology? The last page is a Data Visualization Anatomy Chart. Watch that example’s before-and-after remake in Stephanie’s training on Ignite presentations.

Hot Tip: How can you use the Checklist?

Get in the habit of producing several drafts before sharing final graphs with your clients. Draft, score, edit, repeat!

In this example, we printed an existing graph (page 6 here) in both color and black and white to see how the final chart looked for viewers. We scribbled notes all over the graph and the checklist as we scored. Overall, the graph earned 91% of the possible points—just above the cutoff that enables viewers to read, interpret, and retain the content.

Emery 1

Cool Trick: What’s next for the Checklist?

We are publishing examples to illustrate the 25 items as well as before-and-after remakes. Check out the growing galleries at http://annkemery.com/tag/data-visualization-checklist/ and http://stephanieevergreen.com/tag/data-visualization-checklist/. And we’re taking requests: Which checklist items would you like examples for?

We’re also hoping to present the checklist at the American Evaluation Association’s annual conference in October. Let’s high five there! Please please please can you take a picture of your existing data visualization, apply the checklist, and then take another picture? Tweet your redesigns to @annkemery and @evalu8r. Email them to annkemery@gmail.com and stephanie@evergreenevaluation.com. Fold them into paper airplanes and fly them to us! Send your redesigns and show people how awesome you are!

Big thanks to our pilot reviewers: James Coyle, Amy Germuth, Chris Lysy, Johanna Morariu, Jon Schwabish, David Shellard, Rob Simmon, Kate Tinworth, Jeff Wasbes, and Trina Willard.

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Greetings AEA365! My name is Gretchen Biesecker, and I am the Vice President of Evaluation at City Year. City Year is an education-focused, nonprofit organization founded in 1988 that partners with public schools and teachers to keep students in school and on track to succeed.

This year I competed in my first storytelling slam—an event where people tell five-minute, first-person, true stories. Constructing and telling my story was really fun. I started thinking about new ways our staff at City Year could think about incorporating numbers or data into our communications and reporting.

Lessons Learned:

  • Sharing findings in context is important, especially for audiences that may be unfamiliar with the data. To someone within a school, improving the average daily attendance rate by 2% may be a huge win, but to someone outside education, without context, that increase may sound miniscule.
  • Taking a look at some resources to organize good stories was really helpful to me. Reviewing the ways to construct a good story: 1) Helped me generate ideas for sharing different kinds of numbers and data to share our story; and 2) Emerged as a foundational step before I think about data visualizations and creating reports or presentations.

Rad Resource: Nancy Duarte’s book, Resonate, is now available for free! Duarte shares multiple examples of effective story structures, which may inspire you.

Resonate by Nancy Duarte

Resonate by Nancy Duarte


Hot Tips: Here are some additional ideas you might take from storytelling. You can use numbers to:

  • Create Drama—good stories may be formatted as a sweeping saga, and you can use numbers to convey scale (e.g., City Year serves in 242 schools, reaching 150,000 students). Pairing that with a personal story and results from one child or case is powerful.
  • Set the Stage—numbers can be used to share the problem or give the context for results (e.g., fewer than 40% of students in the nation’s schools score at or above proficiency in English/Language Arts and math).
  • Share the Transformation—good stories have a beginning, middle, and end, or conflict and resolution—so we can share the “before and after” through numbers. You can also “show your work” and the effort or conflict that it took to achieve the results (e.g., in 2012-2013, students in City Year schools spent over 589,100 hours in extended learning time programming).
  • Catch the eye or ear with Repetition—sometimes good stories or speeches have repeating rhymes, words, or numbers, so think about when repeating a particular number may be effective or impactful.

I encourage you to find inspiration and new ideas from the things you love. They may not be within evaluation, but translating them into our field will help us reach more people to put results into action.

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Tony Fujs, Director of Evaluation at the Latin American Youth Center, a DC based non-profit organization. Today, I want to share my experience using R and ggplot2 for data visualization. Ggplot2 is a great tool from the R toolbox (a package, in R lingo). It relies on the powerful Grammar of Graphics framework, which helps “shorten the distance from mind to paper” (Hadley Wickam).

I started using R three years ago, and it has now become my main tool for data analysis and visualization.R is known to have a steep learning curve though, so before getting started, it’s probably a good idea to do a little bit of “cost-benefits” analysis, and check if R is a good fit for you.

Lessons Learned: Benefits

  • More flexibility: R currently meets 99.9% of my dataviz needs. From simple Bar charts, to Maps, to Social Networks… you name it! I can do it directly from R.  No need to learn multiple tools anymore.

Fujs 1

  • Increased productivity:  I often need to generate the same charts on multiple data sets, or multiple subsets of the same dataset. With R it takes almost no effort to do this.

Fujs 2

  • Transparency: Who never did open an Excel file with numbers that seemed to come directly from a magic hat? Coding forces you to become more transparent, and makes your analyses and dataviz easier to replicate. Your colleagues and your future self will thank you for this!

Lessons Learned: Costs

By the way, did I mention that R is free? Your costs-benefits analysis is looking pretty good! You still have some time investment ahead though… In my experience, there are two main barriers when starting dataviz with R and ggplot2:

–       Understanding the Grammar of Graphics: There is a strong logic behind the grammar of Graphics framework. Taking some time to understand it will make your learning experience much smoother.

–       Learning to code: If you don’t have any prior programming experience, moving from a point and click environment to writing code may entail some frustration.

Rad Resources: Here is a list of resources to help minimize learning costs:

1)    Get familiar with the grammar of graphics:

  • Read this paper by ggplot2 creator Hadley Wickam.
  • Check out this slide deck from my dataviz workshop using R and ggplot2. Pay special attention to exercise on slide #25

2)    Get familiar with the R environment:

3)    Start playing with ggplot2

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

We are Rakesh Mohan, Lance McCleve, Tony Grange, Bryon Welch, and Margaret Campbell of the Office of Performance Evaluations, an independent agency of the Idaho State Legislature.

Last year, the Legislature asked us to explain how funds move through the Department of Health and Welfare—the agency with the state’s largest budget. Legislators, including budget committee members, had difficulty understanding the department’s financial information. Given agency complexities and its sheer number of financial transactions (over two million in 2013), no one in recent history had brought together the separate parts of its financial records to explain how the parts function.

Communicating the flow of dollars was the trickiest piece of the study. We considered narratives, tables, and traditional flowcharts. Ultimately, we used Sankey diagrams that helped stakeholders visualize funds moving through the department making them the most useful features of our report.

One diagram shows how dollars flow through a program over a year.

Mohan 1

The second diagram illustrates funds flowing into, out of, and within a program.

Mohan 2

Sankey diagrams depict resources within a system by mapping their flow from an initial set of values to a final set of values. They were especially useful because the lines have widths proportional to their value throughout the system. Legislators could easily see the proportion of each fund throughout the flow. Initially we tried to create the diagrams in Visio, but manually sizing each line was extremely laborious.

Department officials were surprised how clearly our report presented complex aspects of their fund management. They are now using our report for in-house training. They also had us train them on creating Sankey diagrams for future reporting. The feedback from three key stakeholders further illustrates the usefulness of Sankey:

 

“This…really, really was a fine body of work.”

Representative Maxine Bell, Cochair, Joint Finance-Appropriations Committee

 

“The study does much to further enhance the transparency and understandability of the largest budget in state government.”

State Controller Brandon Woolf

 

“[The report] will provide valuable assistance for future planning.”

Governor Butch Otter

Hot Tips:

  1. We used 11×17 paper so the diagram would be big enough to show detail and keep the same orientation as the text.
  2. Subtle colors within similar families were more effective than contrasting colors. We also made colors transparent to clarify intersecting lines.
  3. We wanted to edit text during publishing, so we imported the diagram as an image and added text within the publishing software.

Rad Resources:

  1. We purchased e!sankey, which comes in two versions. The pro version allowed us to link to data in Excel.
  2. Google Charts Sankey
  3. Sankey Diagram History

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Welcome! I’m Ann Emery, Co-Chair of the Data Visualization and Reporting Topical Interest Group (“DVR TIG”). It seems only fitting to tell you more about the DVR TIG through, well, data visualization and reporting! Some highlights:

  • AEA has 48 topical interest groups. They specialize in everything from quantitative methods to advocacy evaluation to data visualization.
  • The Nonprofits and Foundations TIG is the largest, with 1,290 members in 2014. The Lesbian, Gay, Bisexual, and Transgender Issues TIG is the smallest, with 115 members. The DVR TIG falls in the middle, with 821 members this year.
  • 3 out of 4 DVR TIG members are women.
  • Most of the DVR TIG members–92%–have a master’s degree or higher.
  • 28 different countries are represented in the DVR TIG. Roughly 9 out of 10 members are from the United States.
  • The DVR TIG members work in a variety of settings: in nonprofits (33%), private businesses (26%), colleges and universities (26%), federal, state, and local agencies (12%), and school systems (4%).

What about demographic patterns over time? And how does the DVR TIG compare to other TIGs, or to AEA as a whole? This analysis is just the beginning. If you dig deeper into the data, let me know! I look forward to seeing what you find. Have you displayed evaluation findings through a dataviz video like this? How’d it go? I storyboard portions of my workshops and webinars to grab the audience’s attention (especially at the beginning and end of a presentation) and to offer step-by-step explanations of complex charts and diagrams. Do you have questions about how I designed this video? Want to make your own? Share your questions and comments below, or connect with me through @annkemery or annkemery.com. The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.  

No tags

Hey there! I’m Johanna Morariu, a Director of Innovation Network and the Co-Chair of the DVRTIG. DVR is the Data Visualization and Reporting TIG, and we work within the AEA community and through our evaluation work to improve the quality of communications through better data visualization and improved approaches to reporting evaluation findings.

Morariu 1

You could say the first DVRTIG meeting was in 2010, when Stephanie Evergreen convened a small, rowdy group of us to discuss interest in founding the a new TIG to advance issues related to data and information design and reporting within the evaluation community. Since then, we’ve been a fast growing TIG with more than 800 members. (Thanks Stephanie!)

Since those earliest days, TIG members and leaders have worked hard to develop a knowledge and resource base for members and the broader AEA community. One of those newest resources that we’d like to share is a DVRTIG logic model. Yes, our very own logic model!

Morariu 2

Rad Resource: The DVRTIG Logic Model was developed by DVRTIG leadership coming out of the Evaluation 2013 DVRTIG business meeting. The logic model reflects the ideas and ambitions suggested by TIG members at the business meeting. There are more activities than we can hope to achieve, but this is our record of ideas and possibilities for the TIG for the next two years. Check it out and leave a comment in the AEA resource library.

Rad Resource: Another great resource if you’re interested in learning more about the DVRTIG is our website. Anne Worthington is our TIG webmaster and maintains an excellent resources hub of dataviz resources, DVRTIG videos and content, and much more!

Hot Tip: Did you know there is a forum for chatting directly with AEA members interested in data visualization and reporting? Through the DVRTIG website, you can access the DVRTIG eGroup to start a conversation, get feedback on your visualization or report, or share new resources!

We’ve got a great week of posts lined up for you! First, Ann Emery (my colleague at Innovation Network and DVRTIG Co-Chair) will wow you with some stats and visualizations about the DVRTIG. Up next will be Rakesh Mohan and his colleagues from Idaho State’s Office of Performance Evaluations to explain Sankey diagrams. Then we’ll hear from Tony Fujs of the Latin American Youth Center about using R and ggplot2 for data visualization. From there we’ll turn it over to Gretchen Biesecker, Vice President of Evaluation at City Year, to explore storytelling as an effective communication method. And we’ll wrap up the DVR week with Ann Emery and Stephanie Evergreen with a test drive of their Data Visualization Checklist.

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top