AEA365 | A Tip-a-Day by and for Evaluators

CAT | Data Visualization and Reporting

Greetings evaluators! We are Katherine Haugh and Deborah Grodzicki from Innovation Network. At #Eval15 in the Windy City, we conducted a mini-study to try to understand which evaluation approaches evaluators at Evaluation 2015 use most frequently in their work. Drawing on Marvin C. Alkin’s Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences, we re-created the evaluation theory tree to include major evaluation approaches (big green leaves) and evaluation theorists’ names. Take a look at our evaluation theory tree:

Haugh tree 1

The mini-study collected real-time data by asking evaluators to stick a leaf next to the top two approaches they use most often in their work. For those who couldn’t make it to the conference but wanted to participate, we collected votes using #evaltheorytree on Twitter. We had a total of 390 votes and 195 participants. For those unfamiliar with the evaluation theory tree, here is our handout for further explanation and definitions of each evaluation approach.

Marvin C. Alkin and Tina Christie (2004) use the tree metaphor to visually depict foundations (“roots”) from which the field of evaluation emerged, and branches of theoretical work that have grown from these foundations. This study was prompted by our interest in understanding the extent to which these evaluation theories are understood and applied by the practicing community. We recognize that evaluation theorists, academics, and practitioners often work in silos, and for the field to evolve, efforts should be made to increase collaboration. We thought this mini-study would be a fun and creative way to bridge the gap between theory and practice.

What did we learn? (Drum roll, please.)

  1. The “use” branch stole the show. Take a look at the number of votes each evaluation approach received (out of total of 390 votes):

Haugh tree 2

2. Some evaluators use approaches not included on the tree, such as interactive evaluation (2), transformative feminist evaluation (2), genuine evaluation (2), grounded theory (1), collaborative evaluation (1), and culturally responsive evaluation (1).

3. Several evaluators faced existential crises standing before our tree because they were unable to place themselves squarely within one or two approaches. Many commented that they often pull from multiple approaches within one evaluation, so selecting the approach they use most frequently was difficult.

We also we learned that comparing evaluation theories and approaches is useful for identifying and better understanding different perspectives within evaluation, as well as highlighting key debates among prominent theorists. Evaluation theory helps us discern the relative merits of evaluation approaches and improves our frame of reference when choosing an approach. For these reasons and others, we hope to continue bridging the gap between evaluation theory and practice!

We’d love to hear from you! Let’s keep the conversation going with #evaltheorythree or leaf (haha) your comments for us here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

Greetings evaluation enthusiasts! We are Katherine Haugh, Smriti Bajracharya and Kat Athanasiades from Innovation Network.

We have found timelines to be an incredibly useful tool, and believe they are underutilized in the evaluation world. We’d like to change that! For the past few years, we have incorporated timelines into our data analysis process, integrated visual timelines into text-heavy reports, and used them as facilitative tools to increase stakeholder engagement.

Hot Tip: Timelines can be used for data analysis. Visual timelines allow evaluators to see correlations between different events and more easily identify patterns, themes, and trends. This is a rough draft of a timeline that we created for a client, mapping ten years of immigration reform grantmaking. Various contextual elements impacting immigration reform—presidents, major legislative events, and many others—are organized in a visually appealing and clear way. Visual timelines allow evaluators to “zoom out” from a single event so that they can better understand the context in which that event occurred.

Major legislative events

Presidential election cycles:

Haugh timeline 2

External events:

Haugh timeline 1

Rad Resource: The above timeline benefitted from a great deal of formatting and data viz expertise! In Make Your Data Count, we outline the five elements of visual reporting and cover tips for formatting and refining visual report elements—like timelines—in this handout.

Hot Tip: Timelines are an incredibly useful visual addition to a text-heavy report. We created a timeline that looks similar to the one below to complement the aforementioned immigration reform report. This timeline helped us better understand the structural changes that the immigration reform coalition underwent between 2004 and 2007. This timeline also allowed us (and our audience) to visualize and comprehend how and why the coalition evolved over time. Timelines are especially useful for advocacy evaluations in which different players, goals, and resources are constantly shifting.

Hot Tip: Timelines can be used as a tool for increasing stakeholder engagement. For example, in a health advocacy project, community members timelined (yes, a verb!) past successes and challenges in an effort to understand their forward progress around improving health policies. These timelines allowed community members to visualize their work and strategies as a group, helping them interpret their experiences and creating a shared understanding of their anticipated long road to success.

Haugh timeline 3

This timeline’s time periods are shown across the bottom. Red text indicates particularly important events in the time periods.

 

Rad Resource: We presented at AEA2015 in Chicago on “Putting Data in Context: Timelining for Evaluators.” For more tips and tools for creating visual timelines, check out our slides and this handout from the presentation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hello, friends! My name is Yuqi Wang, and I work at Innovation Network. I love figuring out different ways to visualize data, and I want to show you how simple it is to create one of my favorite graphs in Excel—what I like to call the “filled-in bar chart.”

wang 1

  1. Set up your data

List the two time frames in Column A in excel, and label the column “Time frame.” In Column B, enter the two corresponding data points, and label it “Data.”

Hot Tip: This graph works best with percentages, so other than time frames, make sure the rest of your data points are in percentages.

Next: to give our chart the filled-in effect, label Column C “Bottom shading,” and enter the same numbers from Column B in the same order. Then label Column D “Top shading,” and enter the percentage you get when you subtract each of the numbers in Column C from 100%.

Your spreadsheet will look like this:

Wang 2

2. Visualize your data

Highlight the eight data points, and insert a chart. It doesn’t matter which one because we’ll be creating a combination chart. This step is to create a base for our finished graph.

Right click on the chart, and click “Select Data.” Remove all the series under “Legend Entries” because we’re going to enter our own data sets. The prepopulated series will not create the type of graph we want.

Wang 3

Under “Legend Entries,” click on “Add.”

Wang 4

In “Series name”, select the cell that contains the word “Data,” and for “Series values” select the data set under “Data.” Click “Ok.”

 

Wang 5

Next, find the “Horizontal (Category) Axis Labels” section, and click on “Edit.”

Wang 6

In the “Axis label range” box, highlight the two timeframes, and click “Ok.”

Wang 7

Repeat the entire process for “Bottom shading” and “Top shading.”

 3. Reformat your data

After you have your chart, right click on your graph, and select “Change Chart Type”. Under the list of charts, select “Combo,” and within that, choose “Custom Combination.”

“Bottom shading,” “Data,” and “Top shading” data will be displayed, along with the ability to change each data set’s chart.

Choose “100% Stacked Area” chart for “Bottom shading” and “Top shading.” Then choose the “Line with Markers” chart for your “Data.” Click “OK.”

Wang 8

You’re almost done! Delete the numbers you don’t want, delete the legend, and play with the chart size and colors!

Hot Tip: You can create a small multiples chart by copying and pasting multiple filled-in bar charts together!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Elissa Schloesser founder and principal graphic designer at Visual Voice. I specialize in helping organizations visually communicate complex information, concepts and ideas—including evaluation methods, theories and findings.

Rad Resource: Data Stories is a podcast that covers topics on data visualization. I highly recommend it to anyone interested in the field. A recent episode, titled Disinformation Visualization explores the “darker side” of data visualization.

I found this discussion particularly thought provoking and relevant to anyone communicating data. It challenges you to think critically about the data visualizations you create and consume.

Hot Tip: Can you spot a misinformed chart? Below is an example of two charts created from the same dataset.

Schloesser

This example is a little extreme, but I included it to show how data could be manipulated in visualizations.

Both are technically correct, but they strive to tell a different story based on how the data is represented.

Hot Tip: Spot misinformed data visualizations by considering these three things:

  • CONTENT: How was the data was gathered?
    • Before the data is even visualized consider how it was collected.
  • STRUCTURE: How was the data structured or sampled?
    • Does the visualization only represent certain years or a particular age group?
  • PRESENTATION: How was the data presented?
    • Does iconography, colors, annotations, etc. used influence your perception of the data?

Lessons Learned: Think of data visualizations as “visual arguments” rather than “visual evidence”.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor. It’s Independence Day in the US, the anniversary of the day our Declaration of Independence from Great Britain was signed in 1776. As I thought about what topic to tackle for this week’s post, I wondered what insights a word cloud created from the text of the Declaration of Independence might yield. Since I’m familiar with two word cloud generators, I tried each of them.

Here is what I created in Worldle:

Dec of Ind wordle

And here is what I created in Tagxedo:

Dec of Ind Tagxedo

Both generators allow for some creativity with regard to color, and font. Wordle allows you to choose directionality – whether you want all text horizontal, all vertical, or some mix of the two. Tagxedo is best known for its shapes, allowing you to choose from a bank of available shapes that can help you illustrate a point. Tagxedo can also use custom fonts from popular font sites.

Lesson Learned: Word clouds can be fun and have a number of appropriate applications. Use them with caution, however, as they never substitute for analysis, and can potentially detract from important themes. Stuart Henderson and Eden Segal tackle word clouds in a chapter on qualitative data visualization in a recent issue of New Directions for Evaluation.  “Word clouds create a dramatic visual, which likely accounts for their popularity,” the authors posit. “Despite concerns with word clouds, their ease of creation and striking visuals make them a useful tool for evaluators if they are used sparingly and their challenges are acknowledged” (p. 57-58). 

Hot Tip: Henderson and Segal also mention another word cloud generator, TagCrowd, and I decided to give that one a try as well. Here is the result:

Dec ind TagCrowd

While TagCrowd has fewer opportunities for creativity, it does give the user the option to see word counts after each word. I simply checked this option as I created this one.

Rad Resources: A few other aea365 authors – Susan Kistler, Stacy Carruth, Sue Griffey, Jaquelyn Christensen, and Sarajoy Pond – have also shared their perspectives on word cloud generators.

Did you know there is an advanced version of wordle? Learn about this and other Wordle tips, such as how to keep word phrases intact, and how to get numbers to show in Wordle here!

Read the entire article Visualizing qualitative data in evaluation research, by Stuart Henderson and Eden Segal, to understand more about qualitative data visualization.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello! I’m Kate Tinworth, Program Co-Chair of the DVRTIG. I am excited to write about one of my favorite parts of my work as an evaluator— drawing.

Last year at AEA I co-presented a session called, “Drawing Them In: Graphic Facilitation & Evaluation to Strategically Visualize Change” with my friend Chris Chopyak (a rock star who literally wrote the book on using visuals to help businesses address challenges and create strategies). We reminded our audience that we all think visually, images are key to memory and learning, and that you—yes, you—can draw. And you should! Visuals can resolve ambiguity, cut across language and cultural divides, help findings become more salient, and kick start action.

Hot Tip: Find a Local Friend.

If you want to explore the potential of teaming with a graphic facilitator, it’s time to make some new friends. Check out the International Forum of Visual Practitioners (IFVP): http://ifvpcommunity.ning.com/. There’s bound to be someone near you!

Rad Resource: Go to a Class/Workshop.

There are great opportunities to try your hand at graphic facilitation, whether you plan to incorporate it into your evaluation practice or just want to stimulate your visual thinking. Though it can feel intimidating, I highly recommend signing up for a class. Learning graphic facilitation techniques have helped me to sketch out graph and chart ideas, think through report layouts, and get far more creative with methodology and instrumentation.

Cool Trick: Drink.

Some graphic facilitation practitioners, including Chris, do “drink and draws” where you can get some drawing practice over a cocktail. Amazing!

Hot Tip: Draw. All the Time.

To become more comfortable with drawing I draw, all the time. Try covering your dining room/kitchen table with butcher paper and put crayons or colored pencils out. When you sit down for coffee or a meal, draw. Tape paper to the wall and “live capture” TED talks or your favorite podcasts. Carry a notebook and favorite pen everywhere. Commit to drawing for just 2 minutes a day.

Tinworth 1

Cool Trick: Apps.

More of a techie? Draw on your tablet! I like Notability and iMotion.

Rad Resource: Get Inspired.

The DVRTIG is a great place to find inspiration and make connect with colleagues who care about visual thinking. Check out the AEA365 blog posts, the TIG website, and resources on p2i.

Hot Tip: Go Visual.

Whenever you can, try to “go visual” in your projects. Try a visual logic model. Engage stakeholders in drawing. I often get my stakeholders to draw during a training or as I present preliminary data. Lately I have also been experimenting with data placemats, which I learned about through AEA (thanks @VeenaPankaj).

Tinworth 2

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Angie Ficek and I am a Program Evaluator at Professional Data Analysts, Inc. (PDA), a small firm in Minneapolis, MN that specializes in public health evaluation. It wasn’t too long ago that I was reviewing a co-worker’s report and came across a bunch of tables of descriptive statistics like the one below. I couldn’t help but think, “There has got to be a better way to visualize these!”

Ficek 4

Then I remembered learning about dot plots from Ann Emery (for a quick 5 minute tutorial on how to create them, click here) and reading a blog post from Stephanie Evergreen on dumbbell plots. I applied the same concepts and came up with this:

Ficek 5

It’s similar to a box or stock plot, but is actually created with an Excel scatterplot. In this example, I had three x-values (for the min, mean, and max), so I inserted three y-values which tell Excel where the data points should fall on the invisible y-axis.

Ficek 6

As for the oh-so-important formatting of the data points, I added a marker for the mean and a light gray line to signify the range. I added the automated data labels above each data point and formatted the font size and color so that the means stood out more than the min and max since they were the most important value. Then I manually added text boxes to label the min, mean, and max along the top, and the program names along the left side.

I was really excited to find a simple, visual way for our client to see the pattern of these descriptive statistics a little more quickly. I cannot wait until I get to chart some descriptive statistics again! Said no one ever – until now.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Angie Ficek and I am a Program Evaluator at Professional Data Analysts, Inc. (PDA), a small firm in Minneapolis, MN that specializes in public health evaluation. Every once in a while I include timelines in my reports and/or presentations. I know there are many tools out there to create timelines, including Office Timeline and Timeline Maker, but recently I have been using a standard line chart in Excel. Say what?! Yes, I know, it’s crazy! But look at this lovely timeline!

Ficek 1

So how do you do it? Like I said, you will want to use a line chart in Excel. For this example, here is what my datasheet looks like:

Ficek 2

The first two columns are the year and month, and then I have a column for each timeline event – Intake data, Follow-up data, and Reports. Since I want these three events to appear in that order, from top to bottom, I typed in a 3 for each month for which there is Intake data available. I did the same for the Follow-up data with the number 2. And then for the Reports, since these happen at one point in time versus being continuous, I typed a 1 in the month in which a report is submitted, but left the cells in between blank. The exact numbers that you use aren’t as important as having equal spacing and the right amount of spacing between the events. For example, 2.5, 1.5 and .5 would have worked just fine too, but 3, 1.5, and 1 would have resulted in uneven spacing between events, which can be distracting to your audience and personally makes me twitch.

Ok, so our data is set up, and our chart looks something like this:

Ficek 3

Not quite to the final product, but you’re close. From here we delete the vertical axis (and adjust the axis range if needed), the gridlines, and the legend, then reformat each data series. Change the marker type for the Reports to a sunburst, or circle, and adjust the color. Get rid of the markers for the Intake data and Follow-up data series and adjust the line color and width. Then, one at a time, select the first and last data points of these two series to add a sunburst, or circle, to the start and end of them. The final step is to label each event as needed with a text box.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hi! We are Sara Vaca, independent consultant (EvalQuality.com) and Creative Advisor of this blog and Joseph Barnes (Senior Partner at ImpactReady). Today we are going to talk about the work ImpactReady is sponsoring to develop new ideas for using creativity and Dataviz to celebrate the International Year of Evaluation.

Like many evaluators, we have been really inspired by the energy and events that have circulated around EvalYear, and we wanted to find a way to make our contribution to coming-together. After exploring several opportunities, we decided to combine our love of sharing the things we have learnt about practicing evaluation with exploring the exciting new frontier of visualization.

Increasing access to broadband and awareness of how important user experience (UX) is to utilization means that there is a growing interest in Data Visualisation (Dataviz) in the evaluation community. But few of us are yet practicing it as much as we would like to. To help things along, we are attempting to create one new example of evaluation visualization each month throughout EvalYear.

With each of these products, we are seeking to achieve three aims:

To promote the dialogue within the evaluation community;

To foster deeper knowledge of evaluation theories; and

To have fun learning.

Rad Resources: Our first releases include:

  • Evaluation Metromap: Joseph put together this ‘underground map’ of evaluation where each line gathers the most representative of various evaluation aspects such as theories, approaches, techniques, actors, paradigms, etc.
  • What type of evaluator are you? Test: we created this non-scientific multiple-choice test to help evaluators discover if they embrace the paradigm they think they do. We invite you to try it (and help us improve it)!
  • What is good (and bad) evaluation (infographic): after having done many meta-evaluations, we have seen many common mistakes – including in our own work. We created this infographics as a way to provoke ourselves, and our fellow evaluators, to be as clear as possible about what we think ‘good’ really looks like.
  • Ethical issues in evaluation: Ethics is often a forgotten element in evaluation. We wanted to highlight typical controversial moments in evaluation to acknowledge them.

VacaBarnes

Rad Resources: You are welcome to visit the full project at EvalYear.com, and we have more releases coming soon. Here is a sneak peak:

  • A decision tree for helping with the Evaluation design
  • A motion picture explaining what evaluation really is.
  • A Gender dashboard
  • An evaluation coffee table e-book
  • And others!

Keep tuned to know more, and as usual, feel free to comment and contribute with your thoughts to help us learn more!

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hi! I’m Mandi Singleton, Research Assistant at Carson Research Consulting (CRC) in Baltimore, Maryland. Last October I attended Eval 2014 and gave a poster presentation on the process, results and implications of translating data findings into an infographic. My main focus when designing the poster was graphical content. I wanted to impress upon viewers the impact that data visualization can have on audience perception, recall and retention of information. My overall goal was to encourage people to think about the benefits of translating findings into an infographic.

Hot Tips: Below are the steps I took to design my poster:

Singleton 1

Step 1: Identify the basics.

Determine the purpose, audience and key message of your poster. These things will help inform the content of your poster.

Step 2: Determine content.

All content should be supportive of your key message and catered to your audiences’ level of knowledge. Be mindful of terminology, acronyms and icons used in your poster.

Singleton 2

Step 3: Sketch your design.

I sketched the layout for my poster using pencil and paper, but you can use whatever medium you like. The goal here is to get an idea of how you want your poster to look.

Singletn 3

Hot Tip: Sketch your design on a paper that is the same size your poster will be. This helps with spacing and determining the placement of content.

Step 5: Create!

My poster was created using Microsoft Publisher 2013.

Insert your content. Most of my visuals were directly inserted into Publisher as image files. You can also create charts/data visuals in another program (like Excel) and paste them straight into Publisher.

Guide viewer’s eyes by creating visual paths of interest. This can be achieved through the use of color, varied font types and strategically placed shapes.

Singleton 4

Determine the color scheme. My posters color scheme was orange, gray, burgundy and blue. Orange and gray were chosen because they’re our company colors; burgundy and blue were selected because they looked nice with our company colors.

Singleton 5

Hot Tip: You can apply color schemes in Publisher; create your own or select a pre-made one.

Rad Resource(s): For help selecting a color scheme, visit Adobe Kuler (provides RGB & HEX codes) or HailPixel (provides HEX codes).

Step 6: Prepare poster for printing.

Before printing, check for things like spelling errors, objects in non-printable regions and poor image resolution.

Hot Tip: If you’re using Publisher you can run ‘Design Checker’ (found on the home tab) to find problems in your poster for you.

Do you have any other tips for creating a poster presentation? Share by leaving a comment below!

The American Evaluation Association is celebrating Data Visualization and Reporting (DVR) Week with our colleagues in the DVR Topical Interest Group. The contributions all this week to aea365 come from DVR TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Older posts >>

Archives

To top