AEA365 | A Tip-a-Day by and for Evaluators

CAT | Data Visualization and Reporting

We are Lauren Baba and Carol Cahill with the Center for Community Health and Evaluation (CCHE), part of Group Health Research Institute (GHRI) in Seattle. Our team of consultants works with a various stakeholders to evaluate community health initiatives, clinic-community linkages, and health improvement coalitions. We are always brainstorming new ways to make evaluation reports creative, accessible, and interesting for community partners. Drawing on the team’s 20 years of experience, we have developed reporting recommendations to help evaluators write appropriately for different target audiences and clients.

Hot Tip: Synthesize, don’t just summarize. Clients and community partners are looking for your insights into what evaluation findings mean for their work and communities. Refer to Michael Quinn Patton’s AEA365 tips to help focus your synthesis. Then, captivate audiences early; do not wait to answer the “So what?” question at the end of your report. See CCHE’s executive summary for the Sierra Health Foundation as an example.

Hot Tip: Short reports can pack a bigger punch. Right-size the length and level of detail you share with different audiences. If you write a full-length report, consider pulling together tailored briefs for each target audience’s interests, similar to CCHE’s issue briefs for the National Leadership Academy for the Public’s Health.

Hot Tip: Break the mold and write reports that do not follow the usual order of contents or formatting. Be responsive to your client’s expectations for the report, but try to open their eyes to alternatives:

  • Respect your readers’ time and begin with your key takeaway message, then provide the supporting information.
  • Move methodology sections to the end of your report. Or instead include them in a technical appendix with more detailed discussion and data tables for interested readers.

Rad Resource: Check the reading level of your reports and use plain language guides like GHRI’s Program for Readability In Science & Medicine (PRISM) resources to help you write using language that your audiences will understand.

*Look for Part 2 of our article tomorrow!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello! We are Monica Hargraves and Miranda Fang, from the Cornell Office for Research on Evaluation.  We presented together at Eval2012 would like to share some practical tips on literature searches in the context of evaluation.

2016 Update: Monica Hargraves is now Associate Director for Evaluation Partnerships at the Cornell Office for Research on Evaluation; Miranda Fang is now Manager, Development Strategy and Operations at Teach For America – Los Angeles

Program managers often face an expectation worthy of Hercules: to provide strong research-quality evidence that their program is effective in producing valuable outcomes. This is daunting, particularly if the valued outcomes only emerge over a long time horizon, the program is new or small, or the appropriate evaluation is way beyond the capacity of the program.  The question is, what can bridge the gap between what’s feasible for the program and what’s needed in terms of evidence?

Hot Tip: Strategic literature searches can help. And visual program logic models provide an ideal framework for organizing the search process.

Quoting our colleagues Jennifer Urban and William Trochim in their AJE 2009 paper on the Golden Spike,

The golden spike is literally a place that can be drawn on the visual causal map … where the evaluation results and the research evidence meet.”

We use pathway models, which build on a columnar logic model and tell the logical story of the program by specifying the connections between the activities and the short-term outcome(s) they each contribute to, and the subsequent short- or mid-term outcome(s) that those lead to, and so on.  What emerges is a visual program theory with links all the way through to the program’s anticipated long-term outcomes.

The visual model organizes and makes succinct the key elements of the program theory. It helps an evaluator to zero in on the particular outcomes and causal links that are needed in order to build credible evidence beyond the scope of their current evaluation.

Here’s an example, from a Cornell Cooperative Extension program on energy conservation in a youth summer camp.  Suppose the program needs to report to a key funder whose interest is in youth careers in the environmental sector. If the program evaluation demonstrates that the program is successful in building a positive attitude towards green energy careers, then a literature search can focus on evidence for the link (where the red star is) between that mid-term outcome and the long-term outcome of an increase in youth entering the green workforce.

The American Evaluation Association is celebrating Best of aea365, an occasional series. The contributions for Best of aea365 are reposts of great blog articles from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I am Jonathan Jones, Senior Monitoring and Evaluation Technical Advisor at CAMRIS International. I am also co-chair of AEA’s International and Cross Cultural TIG.

As an evaluator, have you ever gotten feedback from someone that changed the way you write evaluation reports? Someone* gave me this great feedback several years ago on a draft evaluation report I had written – assume that the report reader does not know anything about the program that is being evaluated. Therefore, it is important to include (concise) information about key milestone programmatic events and important contextual events that might have influenced the program – ideally in a graphic.

From that point on, I have included a timeline graphic in the introductory section of my evaluation reports. This graphic is always well received by the client. Not only does the graphic tell the story of the program in one picture, program staff tend to really like it because it shows that the program did not take place in a vacuum – that there were typically significant contextual events that were influential.

Rad Resource: I use Office Timeline to create the timeline graphic (although there are many other software solutions available). Office Timeline is a simple, easy to learn, PowerPoint plug in. The free version works well, but I found that I quickly bumped up against the limitations, and needed the functionality of the paid version. Below is an example of a timeline graphic created with Office Timeline (note, the information in the graphic is fake).

Jones

Hot Tip: I like to include contextual milestones above the timeline, and program milestones below (see example above).

Hot Tip: Ask the implementer/program staff to identify the key context and programmatic milestone events for you! This is a great way to involve the implementer in the evaluation process. I always send the implementer a simple table to fill out.

Hot Tip: Include the evaluation (and any prior evaluations) as milestone program events. This can help the reader situate the evaluations conducted for the program within the wider context.

Hot Tip: Less is more! It can be tempting to include every event that might have been important, but the graphic can quickly become cluttered, and the story of the program lost. Work with the implementer to prioritize the key events that must be included. I also don’t include actual dates of key events, with the idea that the reader will not remember key dates, but will want to remember the overall story.

*My thanks to Tessie Catsambas, CEO/CFO of EnCompass LLC who gave me this important feedback several years ago on an evaluation report I wrote.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Our names are Marc Wheeler and Salem Valentino and we are internal evaluators for Big Brothers Big Sisters of America.

There is a lot of buzz today about infographics.  Many of you may have thought about using infographics in your evaluation reports to try to translate your findings more effectively.  We recently took the plunge and incorporated infographics in our 2013 Youth Outcomes Report .

For example, we created the infographic below to clarify the theoretical connection between the short-term outcomes we currently measure for each youth and those long-term outcomes of interest to many stakeholders. The graphic summarized a large quantity of research literature in a single infographic that was easily interpretable and concise.

Wheeler 1

Lessons Learned:

  1. High-quality infographics require a certain level of expertise; as we didn’t have the relevant experience in house, we contracted with an external graphic designer who delivered great results.
  2. Beyond experience, it also takes time to develop your infographics and get them right.  First, you need to figure out the story behind your data.  Then, what are the best ways to illustrate this story, while remaining true to the data?  For instance, we wanted to communicate effect sizes but didn’t want to take up space in our report explaining what they meant to a lay audience.  So instead, we developed the visualization below to better illustrate our story of the magnitude of youth outcomes.  Lastly, infographics require a number of iterations and can benefit from the input of diverse audiences.  Budget your time accordingly.

Wheeler 2

  1. Understanding your project needs will help you choose the right designer. We looked at several resources on the internet to find a graphic designer. For us it was important that our work with the designer was collaborative; we wanted to ensure the quality of the evaluation content. Due to our timeline, it was also important that our designer could design the entire report and not merely the infographics.  You may also want to ask your designer how comfortable they are with Excel or other types of data you will be using in the report.

Rad Resources:

Elissa Schloesser at Visual Voice – our designer’s 5 Steps for Translating Evaluation Findings into Infographics

Visual.ly’s Marketplace service will find a designer for you and help you create an infographic for one price.

Easel.ly is a website where you can create your own infographic for free.

The American Evaluation Association is celebrating Best of aea365, an occasional series. The contributions for Best of aea365 are reposts of great blog articles from our earlier years. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi! I’m Kevin Lee. One key takeaway from my year as a GEDI scholar is the idea of communication in evaluation. More specifically, I was exposed to thinking about how to communicate findings in meaningful ways. Whereas writing a grant, working on a manuscript, or completing a report template can often lead to page after page of convoluted text and technical terminology, conveying meaningful data and results has the potential to be less structured yet equally powerful. In reflecting on my experience as a GEDI, I share some guidelines that have helped me think about how I communicate evaluation findings:

  1. Understand and define the stakeholders. It’s important to carefully consider for who the findings are intended. This dictates the kind of information that an evaluator wants to communicate and the most appropriate way to do this. Whether the results are for government officials, academic researchers, funders, or community members makes a difference.
  2. Determine what data are important. As any evaluator knows, an evaluation can generate an extensive amount of data. It’s often necessary to consider what kind of information is relevant based on the evaluation purpose and its stakeholders. While it may be tempting to share all the results, this may be unnecessary for stakeholders who may become overwhelmed by information.
  3. Interpret the data clearly. Data can be interpreted in different ways. Think about what the data are saying and how this can be communicated clearly, concisely and in a way that is relevant to your audience. Reported data should have a purpose and tell a story rather than simply list numbers and facts.
  4. Think about design. To effectively share evaluation findings, consider how the data should be presented. Not only does this refer to interpretation, but also in thinking about the presentation format, data visualization, graphic use, and layout composition, among other aesthetic elements. Creativity and design can garner attention, generate interest, and promote memory retention.

Rad Resources: Dr. Stephanie Evergreen is always insightful about data presentation. You can find information about her books and workshops here: http://stephanieevergreen.com/.

Try your best to enroll in her pre-conference sessions at the annual AEA conference as they tend to fill quickly.

Also very helpful are Elissa Schloesser’s artful interpretations of data. See more here: http://myvisualvoice.com/.

The American Evaluation Association is celebrating Graduate Education Diversity Internship (GEDI) Program week. The contributions all this week to aea365 come from AEA’s GEDI Program and its interns. For more information on GEDI, see their webpage here: http://www.eval.org/GEDI Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Sheena Horton, Senior Analyst at MGT of America, Inc., and Board Member for the Southeast Evaluation Association (SEA). It is no surprise that the use of data visualization in reporting and marketing is thriving. Studies have shown that humans process visual information better and faster than text. Data visualization can provide viewers with easily understood, actionable data and be more engaging to an audience in instances when the use of simple text can fall short.

Lesson Learned: Extend your use of data visualization beyond the workplace, and apply your data design skills to evaluating and strengthening your professional network, skills, and career profile.

Hot Tip: Conduct a social network analysis to evaluate your professional social network to identify your strong connections. Look for areas in your field, specialization, geographic location, or position type and level (e.g. managerial or mid-level) where you may need to build better connections. Note the networks where you can make contributions, and identify the best connections for conducting outreach to learn more about a specific area or skill.

Rad Resources: There are numerous data visualization tools available online to help you get started analyzing your social network. Socilab can provide you with a high-level overview of your LinkedIn network connections to jumpstart your analysis.

Hot Tip: Use data visualization to take stock of your hard and soft skills to determine the range of your strengths and to pinpoint skills to develop. A simple mind maphttp://www.mindmapping.com/ of your core skills can help you see where you can build upon your current skill set, or discover new skill areas to develop. Mind mapping adds focus to your professional development brainstorming, and helps to initiate an action plan.

Rad Resource: MindMeister is a popular and user-friendly mind mapping tool that can help you to start charting your skills quickly.

Hot Tip: Include data visualization in your resume, on LinkedIn, or on your professional website to showcase your skills and your career through visual storytelling. Determine where using data visualization can be useful based on your audience and the message you want to communicate. The visualization you select should display your data appropriately and engage your audience. A minimalist design works best; be careful not to go overboard. The key is to communicate your data simply and quickly.

Rad Resources: ResumUP and Vihttp://vizualize.me/zualize.me are good starting resources to experiment with framing your data and gathering ideas for display. A visualization that works for one person’s data may not work as well with your own.

The American Evaluation Association is celebrating Southeast Evaluation Association (SEA) Affiliate Week with our colleagues in the SEA Affiliate. The contributions all this week to aea365 come from SEA Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings evaluators! We are Katherine Haugh and Deborah Grodzicki from Innovation Network. At #Eval15 in the Windy City, we conducted a mini-study to try to understand which evaluation approaches evaluators at Evaluation 2015 use most frequently in their work. Drawing on Marvin C. Alkin’s Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences, we re-created the evaluation theory tree to include major evaluation approaches (big green leaves) and evaluation theorists’ names. Take a look at our evaluation theory tree:

Haugh tree 1

The mini-study collected real-time data by asking evaluators to stick a leaf next to the top two approaches they use most often in their work. For those who couldn’t make it to the conference but wanted to participate, we collected votes using #evaltheorytree on Twitter. We had a total of 390 votes and 195 participants. For those unfamiliar with the evaluation theory tree, here is our handout for further explanation and definitions of each evaluation approach.

Marvin C. Alkin and Tina Christie (2004) use the tree metaphor to visually depict foundations (“roots”) from which the field of evaluation emerged, and branches of theoretical work that have grown from these foundations. This study was prompted by our interest in understanding the extent to which these evaluation theories are understood and applied by the practicing community. We recognize that evaluation theorists, academics, and practitioners often work in silos, and for the field to evolve, efforts should be made to increase collaboration. We thought this mini-study would be a fun and creative way to bridge the gap between theory and practice.

What did we learn? (Drum roll, please.)

  1. The “use” branch stole the show. Take a look at the number of votes each evaluation approach received (out of total of 390 votes):

Haugh tree 2

2. Some evaluators use approaches not included on the tree, such as interactive evaluation (2), transformative feminist evaluation (2), genuine evaluation (2), grounded theory (1), collaborative evaluation (1), and culturally responsive evaluation (1).

3. Several evaluators faced existential crises standing before our tree because they were unable to place themselves squarely within one or two approaches. Many commented that they often pull from multiple approaches within one evaluation, so selecting the approach they use most frequently was difficult.

We also we learned that comparing evaluation theories and approaches is useful for identifying and better understanding different perspectives within evaluation, as well as highlighting key debates among prominent theorists. Evaluation theory helps us discern the relative merits of evaluation approaches and improves our frame of reference when choosing an approach. For these reasons and others, we hope to continue bridging the gap between evaluation theory and practice!

We’d love to hear from you! Let’s keep the conversation going with #evaltheorythree or leaf (haha) your comments for us here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

Greetings evaluation enthusiasts! We are Katherine Haugh, Smriti Bajracharya and Kat Athanasiades from Innovation Network.

We have found timelines to be an incredibly useful tool, and believe they are underutilized in the evaluation world. We’d like to change that! For the past few years, we have incorporated timelines into our data analysis process, integrated visual timelines into text-heavy reports, and used them as facilitative tools to increase stakeholder engagement.

Hot Tip: Timelines can be used for data analysis. Visual timelines allow evaluators to see correlations between different events and more easily identify patterns, themes, and trends. This is a rough draft of a timeline that we created for a client, mapping ten years of immigration reform grantmaking. Various contextual elements impacting immigration reform—presidents, major legislative events, and many others—are organized in a visually appealing and clear way. Visual timelines allow evaluators to “zoom out” from a single event so that they can better understand the context in which that event occurred.

Major legislative events

Presidential election cycles:

Haugh timeline 2

External events:

Haugh timeline 1

Rad Resource: The above timeline benefitted from a great deal of formatting and data viz expertise! In Make Your Data Count, we outline the five elements of visual reporting and cover tips for formatting and refining visual report elements—like timelines—in this handout.

Hot Tip: Timelines are an incredibly useful visual addition to a text-heavy report. We created a timeline that looks similar to the one below to complement the aforementioned immigration reform report. This timeline helped us better understand the structural changes that the immigration reform coalition underwent between 2004 and 2007. This timeline also allowed us (and our audience) to visualize and comprehend how and why the coalition evolved over time. Timelines are especially useful for advocacy evaluations in which different players, goals, and resources are constantly shifting.

Hot Tip: Timelines can be used as a tool for increasing stakeholder engagement. For example, in a health advocacy project, community members timelined (yes, a verb!) past successes and challenges in an effort to understand their forward progress around improving health policies. These timelines allowed community members to visualize their work and strategies as a group, helping them interpret their experiences and creating a shared understanding of their anticipated long road to success.

Haugh timeline 3

This timeline’s time periods are shown across the bottom. Red text indicates particularly important events in the time periods.

 

Rad Resource: We presented at AEA2015 in Chicago on “Putting Data in Context: Timelining for Evaluators.” For more tips and tools for creating visual timelines, check out our slides and this handout from the presentation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hello, friends! My name is Yuqi Wang, and I work at Innovation Network. I love figuring out different ways to visualize data, and I want to show you how simple it is to create one of my favorite graphs in Excel—what I like to call the “filled-in bar chart.”

wang 1

  1. Set up your data

List the two time frames in Column A in excel, and label the column “Time frame.” In Column B, enter the two corresponding data points, and label it “Data.”

Hot Tip: This graph works best with percentages, so other than time frames, make sure the rest of your data points are in percentages.

Next: to give our chart the filled-in effect, label Column C “Bottom shading,” and enter the same numbers from Column B in the same order. Then label Column D “Top shading,” and enter the percentage you get when you subtract each of the numbers in Column C from 100%.

Your spreadsheet will look like this:

Wang 2

2. Visualize your data

Highlight the eight data points, and insert a chart. It doesn’t matter which one because we’ll be creating a combination chart. This step is to create a base for our finished graph.

Right click on the chart, and click “Select Data.” Remove all the series under “Legend Entries” because we’re going to enter our own data sets. The prepopulated series will not create the type of graph we want.

Wang 3

Under “Legend Entries,” click on “Add.”

Wang 4

In “Series name”, select the cell that contains the word “Data,” and for “Series values” select the data set under “Data.” Click “Ok.”

 

Wang 5

Next, find the “Horizontal (Category) Axis Labels” section, and click on “Edit.”

Wang 6

In the “Axis label range” box, highlight the two timeframes, and click “Ok.”

Wang 7

Repeat the entire process for “Bottom shading” and “Top shading.”

 3. Reformat your data

After you have your chart, right click on your graph, and select “Change Chart Type”. Under the list of charts, select “Combo,” and within that, choose “Custom Combination.”

“Bottom shading,” “Data,” and “Top shading” data will be displayed, along with the ability to change each data set’s chart.

Choose “100% Stacked Area” chart for “Bottom shading” and “Top shading.” Then choose the “Line with Markers” chart for your “Data.” Click “OK.”

Wang 8

You’re almost done! Delete the numbers you don’t want, delete the legend, and play with the chart size and colors!

Hot Tip: You can create a small multiples chart by copying and pasting multiple filled-in bar charts together!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Elissa Schloesser founder and principal graphic designer at Visual Voice. I specialize in helping organizations visually communicate complex information, concepts and ideas—including evaluation methods, theories and findings.

Rad Resource: Data Stories is a podcast that covers topics on data visualization. I highly recommend it to anyone interested in the field. A recent episode, titled Disinformation Visualization explores the “darker side” of data visualization.

I found this discussion particularly thought provoking and relevant to anyone communicating data. It challenges you to think critically about the data visualizations you create and consume.

Hot Tip: Can you spot a misinformed chart? Below is an example of two charts created from the same dataset.

Schloesser

This example is a little extreme, but I included it to show how data could be manipulated in visualizations.

Both are technically correct, but they strive to tell a different story based on how the data is represented.

Hot Tip: Spot misinformed data visualizations by considering these three things:

  • CONTENT: How was the data was gathered?
    • Before the data is even visualized consider how it was collected.
  • STRUCTURE: How was the data structured or sampled?
    • Does the visualization only represent certain years or a particular age group?
  • PRESENTATION: How was the data presented?
    • Does iconography, colors, annotations, etc. used influence your perception of the data?

Lessons Learned: Think of data visualizations as “visual arguments” rather than “visual evidence”.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top