AEA365 | A Tip-a-Day by and for Evaluators

Search

Hi, I’m Craig Wiles, Senior Consultant at Public Sector Consultants in Lansing, Michigan. I provide research and evaluation services for clients in health and human services, education and the environment. I am sharing a tip on how to use Tableau as part of a data exploration process with a group of stakeholders.

To begin, I did the heavy statistical lifting outside of Tableau, so this would not lapse into a data-mining exercise. In this case, I worked with a state-level stakeholder group to identify data sources, research priorities, and statistically significant correlations in the data. Once we had our short list of correlated variables to explore in more detail, we convened a series of two hour, interactive data exploration sessions. At these sessions, we used Tableau to visually display the data (in this case, educational data), identify high and low performing school districts, and look for other obvious patterns or outliers in the data.

We tended to use stacked bar charts and scatter plots to help with this visual part of the data exploration. One tool in Tableau that was especially helpful in this context was the filter bar. Using the filter tool, we were able to adjust the range of scores we were looking for in our combination of variables according to tolerances set by the stakeholders. For example, we looked for school districts that had a high graduation rate, low dropout rate, and a higher ratio of students with disabilities in general education classrooms.

I recommend using Tableau for data exploration because it is:

  • Visual,
  • Interactive, and
  • Builds capacity and ownership.

I could have presented this data in charts and graphs and led a typical ‘sit-and-get’ meeting and landed at the same place (conceptually) at the end of the day. This kind of visual and interactive process, however, really helped to engage my stakeholders, especially those that are usually averse to numbers and data. Ultimately, this was as much about the process as it was about the data.

After our interactive sessions, this group began a series of local focus group conversations with voluntary school districts to further explore the relationships we identified. This qualitative data has provided depth of detail and rich context to the quantitative relationships we explored together.

Using the filter tool:

Tableau filter tool

Using a scatter plot:

Tableau scatter plot

Click to increase size

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

My name is Julie Lo and I’m a research assistant at Public Profit.  We help youth service agencies and educational organizations manage what matters.  I want to share with you a way of presenting survey data through an embedded interactive interface created using the program Tableau Desktop and later shared online using Tableau Public.

We process thousands of stakeholder surveys each year for our clients and were thrilled when one of them approached us about creating a method of publishing a city-wide survey of thousands of high school youth to a wider audience.  We found a way to consolidate what in past years has been a 250-page report to an interactive set of data visualizations:

Emphasizing the many different things to consider when creating any kind of data visualization, I’ve included some practices that we will be sure to consider for future projects like this.

Lesson Learned: Which came first: function or design? 

The first leg of the project involves building out of the basic frame: pages, buttons, selecting the best viz to tell the story. Leave enough fuel in your project’s gas tank to be able to do this. Share it with your client for feedback to make sure that it not only works well, but is also designed in a way that is accessible for youth, teachers, and policy analysts alike!

We worked through several drafts before arriving at the final product both internally and with our client.  This helped us to make decisions around what types of visualizations work best for the data on hand.  Luckily, our client has been working on this project for multiple years and was able to share valuable insights on what information to emphasize for the audience and the “look” that would be most appealing.

Lesson Learned: How fit is your data?

Tableau Public has excellent built-in online sharing features and is seamlessly linked to Tableau Desktop making the work of transferring your work online quick and easy. However, Tableau Public does not have a way to suppress the underlying data, posing potential issues around the confidentiality of individual survey takers.  We got around this by using summarized data that we prepared outside of Tableau in SPSS and used summarized data for the underlying data for the Tableau Public file.  This is an important step to consider when planning your project timeline since it adds a little bit of data-prep work.

Lesson Learned: So many questions, where to begin? 

We worked closely with our client to help develop themes to group survey items together and to select ways to filter data by demographics. This creates further utility for both our clients and those they serve. Presenting questions grouped together by theme can help viewers navigate survey results for their populations of interest, helps to make sense of data and keeps pages free of the dreaded data-viz clutter.

Rad Resources – Related aea365 posts:

Rad Resources:

We’re celebrating Data Visualization and Reporting Week with our colleagues in the DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

·

I’m Susan Kistler, the American Evaluation Association’s Executive Director. I contribute each Saturday’s post to aea365. I’m striving to improve my capacity to use Tableau, a data exploration and visualization tool. I’ve read the manual, watched their video tutorials, will be attending training, and have spent time working with the product on my own. However, as with any complex tool, the devil is in the details and I occasionally find myself caught up with how to make it do exactly what I want as efficiently and effectively as possible.

Today, I wanted to share three tools that I use for support as I strive to use new software and hardware, and to get the most from products with which I have worked for years.

Rad Resource – Product Discussion Lists: Discussion lists, or forums, devoted to a particular product are usually populated by both users and representatives from the manufacturer. They are places to learn from others who have ‘been there, done that.’ A quick check of Tableau’s site and I found Support -> Forums. Support forums tend to be fairly no-nonsense question-and-answer spaces.

Rad Resource – Experts Exchange: Experts Exchange is a site where you can post tech questions on almost any topic or platform and get answers from thousands of other users. It is like having ready access to hundreds of discussion lists covering multiple products. Experts Exchange isn’t free, but for $100 per year, I’ve received multiple specific answers tailored to my problem. I’ve found those who respond to be very sophisticated users who know the details of a particular product.

Rad Resource – Users Groups: Sometimes, you want a bit more than Q&A. You may be seeking demonstrations, meetups, peer-critiques, examples, and networking. Users Groups bring together those with an interest in a particular product and usually supplement what you’d find on a discussion list with other streams for capacity building. User Groups used to be location-dependent, so there were often many groups for a particular product, but increasingly there are virtual groups as well. Apple and SAS have both set up spaces on their website to support users groups for their products. And, if you know of other users groups that may be of interest to evaluators, please share them in the discussion section for this post.

Hot Tip – Join the Tableau Virtual Users Group: I like users groups. They combine product support and demonstration with networking and commiseration. I’m starting a Tableau virtual users group and welcome anyone working in evaluation or a related field who’d like to join us to add to the discussion on AEA’s LinkedIn group here. We’re hoping to meet monthly via webinar with short demos, peer-critique, time for questions and networking, etc.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Nina Potter and I am currently the Director of Assessment for the College of Education at San Diego State University. I’d like to share a little about a tool we are using for data visualization.

One of my responsibilities is to work with program directors and department chairs to evaluate academic programs across the college’s eight departments and 30+ programs. Our programs vary greatly in size and each has its own goals and student learning outcomes. Plus, we have some common goals across the college. We wanted to have a common tool that would allow us to share data across the college, but it had to be very flexible in terms of the kinds of data that it could handle as well as the kinds of reports that it could generate. After a lot of exploring, we chose Tableau.

Rad Resource: Before coming to SDSU, I had never heard of Tableau, in fact I had not heard the term “data visualization tool.” First I will tell you what it is NOT. Tableau is not a tool for data entry. You use Tableau to access data from other data sources such as spreadsheets or databases. This was important because our programs use many different tools to collect data, from electronic portfolio systems to paper and pencil tracking (we do require them to at least put the data in a spreadsheet). And, Tableau does not do advanced statistics; although it does do simple regression and t-tests. For statistical tests, we still use other statistic packages.

So what does Tableau do? Tableau allows you to link into multiple data sources, and quickly and easily create interactive graphs and charts that are updated in real time as your data sources are updated. It has a variety of choices for visualizations such as tables, line graphs, bar charts, pie charts and geographical maps. With just a few clicks you can easily change the type of chart, add colors, add filters and drill down to data that fits certain criteria. The charts are interactive so that anyone viewing the charts can apply filters and view the data they want to focus on.

For example, we have some assessments that are given across multiple programs. We can create a chart that looks at student progress over time and add filters such as program, gender, ethnicity, and age. A person who is evaluating the program as a whole can compare the results from program X to program Y to see if there is equity across multiple demographic groups. Additionally, a person who is working with individual students can download a list of students who have failed more than one assessment in a given program.

Want to hear more about Tableau from Nina? Join her on April 29 for “Data in, Brilliance Out with Tableau” as part of AEA’s Coffee Break Demonstration Series. More information and registration may be found at http://comm.eval.org/EVAL/coffee_break_webinars/Home/Default.aspx. Free for AEA members!

· ·

Hi! I’m Sarah Dunifon, Research and Evaluation Associate at the Wildlife Conservation Society (WCS). In my role, I create many reports and I’m always looking for efficient tools for data visualization. I’ve found a few different programs to display location data, which I’d like to share with the wider AEA community.

Rad Resources:

Google Fusion Tables

Google Fusion Tables is an experimental app add-on that you can link to Google Drive. It allows users to create online, interactive heat maps and feature maps. Privacy settings are managed in the same way as other Google products where users can decide on a range between public and only available to you as the user. Viewers can manipulate the maps in various ways, such as filtering results, scrolling around the map view, or switching between map types.

Feature map of countries with WCS offices – Fusion Tables

Infogr.am

Infogr.am offers quick and easy interactive heat maps which can be shared via weblink. The free version includes a United States map and a world map, and your data will be public, whereas the paid versions have data privacy, more map choices, and the option to download the maps as images.

Heat map of countries with WCS offices – Infogr.am

Excel Apps – Geographic Heat Map

With the “Geographic Heat Map” app on Microsoft Excel, you can create either a world or United States heat map. The data is private and you can save your final map as a picture, making it a good option for inserting into a report. This app doesn’t have much customizability in color and style, but I’ve been able to paste the image into another program (say Microsoft Word or PowerPoint) and edit the image there.

Heat map of countries with WCS offices – Geographic Heat Map with Excel Apps

Tableau

Tableau offers feature maps and heat maps for free, though the data will be public. This program is highly customizable and makes some beautiful visualizations. However, you might find there is a bit of a learning curve to using this software. The visualizations can be saved as an interactive display in “presentation mode” or uploaded to the Tableau Public gallery where they can be shared digitally.

Heat map of countries with WCS offices – Tableau

Excel Powermap

Powermap in Microsoft Excel lets you create private feature maps in a variety of themes on a 3D globe. The map can be an online interactive display or an image produced by taking a screen grab through the program. The screengrab puts a picture of the image onto clipboard, which you can then paste into another program.

Feature map of countries with WCS offices – Excel Powermap

Hot Tip: Consider how you intend to use the map before you start building it. If it needs to be interactive, choose an online format. If it needs to be put into a report, pick a program with capabilities to export a high-resolution image, rather than just a screenshot.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

We are Ana Flores and Joshua Paul at Volunteers of America – Los Angeles. At the AEA Evaluation 2016 conference, we presented a panel entitled “A Picture is Worth a Thousand Words…But Will They Use It?”. Today, we want to provide additional information regarding how to make data more user-friendly.

The Evaluation Department at Volunteers of America – Los Angeles (VOALA) is tasked with providing evaluation services as needed to more than 70 social service programs. Staff in these programs are dedicated to helping people and many find data unappealing. Addressing communication barriers has given us the opportunity to learn a number of lessons.

Lessons Learned #1: Needs Change, Open a Dialog with Succinct Visualizations  

Understanding stakeholder needs and how they fit into a program model is a major part of any evaluator’s task. Unfortunately, we have found that stakeholder needs and program models can change rapidly, and stakeholders do not always volunteer information about these changes.

We were once mystified as to why one of our programs — whose initial purpose was to connect with and refer homeless veterans to local services — had such poor monitoring results. Traditional reporting methods failed to open a dialog that could bring the core problem to light. After months of discussion, we tried a new visualization-based design (see image) that demonstrated the discrepancy between the goal and present performance and prompted program leadership to identify the issue. The staff had been focused on the transportation of clients to appointments, a secondary program activity, which had not originally been designated as important to track.

internal-eval-image-1

internal-eval-2

Lessons Learned #2: Only Show What You Need to Show

 

Past reports for many of our programs provided detailed data, presenting every single outcome for individual clients. However, this level of information was not necessary for program performance discussions and was a distraction from the overall outcomes included in the report. Using Tableau, the detailed information was removed, and only overall outcome percentages and targets were kept on the graph. With outcomes presented this way, VOALA upper management was able to get the information they needed to make program recommendations and help program directors implement better practices.

 

Lessons Learned #3: Use Interactivity

Giving your audience an opportunity to control the data makes it easier for them to make inferences about the information. Visual analysis programs, like Tableau, allow us to provide interactive reports so that upper management and program directors can filter results by key demographics or periods of time, depending on what is useful to them.

 

Having these types of “quick snapshot” visualizations has helped upper management at VOALA communicate recommendations with programs.

Experimenting with these different data visualization techniques has improved our discussions with key staff, helping us ask hard questions while reducing staff resistance to data. Otherwise, the response to “Why is this benchmark never reached?” might just be silence.

The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi! We are Laura Beals, Director, and Barbara Perry, Evaluation Manager, of the Department of Evaluation and Learning at Jewish Family and Children’s Service Boston, a multi-service nonprofit in Massachusetts. At Eval 2015, we learned about “Data Placemats” from Veena Pankaj of the Innovation Network. Recently, we held several placemat-focused “Learning Conversations” with one of our multi-program divisions. We created seven placemats for these meetings:

  1. An overview of the Learning Conversation and placemat process.
  2. Client census—new and active—over the past four years for each program.
  3. Client demographics by program.
  4. Client geographic distribution heat map. This placemat was interactive, using Tableau. We wanted not only to show the geographic distribution of clients in Massachusetts, but also to provide an opportunity to explore the data further, through the use of filters for program and key demographics.
  5. A network analysis showing referral sources.
  6. A network analysis showing how clients were served by multiple programs at the agency.

(click for larger image)

beals

7. A learning and dissemination plan. This placemat encouraged meeting participants to use the data and allow our team to     create specific follow-up documents and undertake follow-up analysis.

Lessons Learned:

  • During the planning stages, check-in with stakeholders from around the organization. We asked the program director, division director, grant writers, and development associates what they wanted to learn about the division. Their responses allowed us to tailor the placemats to be as useful to as many people as possible.
  • Don’t forget to include the staff! In order to share the placemats and get feedback from the direct-service staff, at an all-staff meeting we held a shorter placemat discussion, focusing on two placemats; the other placemats were provided for later review. We also hung up the placemats near the staff offices and provided sticky notes for feedback and observations.
  • Be ready to “go on the road” with your placemats. We found that word spread about our placemats and there was interest from various stakeholders who had not been able be part of the original few meetings. By continuing the conversations, we were able to increase learning and generate new ideas.
  • Bring data chocolates! We had been waiting for an opportunity to create data chocolates, after being inspired by Susan Kistler. We wrapped shrunken versions of several of the graphs around chocolates. They put everyone in a good mood to talk data—the lightheartedness of our gesture helped break down barriers and were a great conversation starter.

Rad Resources:

The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! Our names are Natalie Wilkins and Brandon Nesbit and we are both evaluators at the Centers for Disease Control and Prevention (CDC), in the National Center for Injury Prevention and Control (NCIPC).

One of the projects we provide evaluation support for is the Injury Control Research Centers (ICRCs) program, funded through NCIPC. This has provided us with a number of important lessons learned around evaluating multi-site research center programs that are engaging in translational research and outreach.

There are 10 ICRCs across the country, funded to conduct innovative research on the prevention of injury and violence.  These institutions serve as training centers for the next generation of injury and violence prevention researchers and act as information centers on injury and violence prevention for the public.  ICRCs are also pioneering innovative approaches to the translation of research to practice. They conduct translational research studies and engage in a variety of outreach activities to translate research on evidence-based injury and violence prevention strategies into practice settings. For example, one of the ICRCs works with partners to assess their capacity for using research findings in their work, and then provides tailored technical assistance based on each partners’ specific needs to ensure research is translated into practice.  In addition to these “research to practice” activities, some ICRCs are also employing a “practice to research” approach to their translational research, leveraging their outreach activities and partnerships in the field to inform their research priorities.

As evaluators of this comprehensive, multi-site research center program, one of our challenges was to show the impact of the ICRCs’ translational research and outreach activities on bridging the gap between research and practice. To this end, CDC and the ICRCs developed a set of indicators to capture information on impact (e.g. studies, partnerships, outreach activities, development of research and practice tools, etc.). We display data on these indicators through Tableau, software that allows users to analyze, visualize, and share data in an interactive way.

Hot Tip: Visually presenting evaluation data through interactive dashboards allows stakeholders to glean their own insights while still ensuring key messages are communicated.  Tableau enables us to showcase the approach and impact of each of these unique research centers, while also providing the option of presenting a “bird’s eye view” of the impact of the entire ICRC program as a whole.

Wilkins & Nesbit

Lesson Learned:  Translational research and outreach can take many forms. Engage your stakeholders in the evaluation process early so you can ensure they have a clear understanding of the kinds of information you are looking for.

Rad Resource: For more information on how evaluators have used Tableau, check out the AEA365 archives- http://aea365.org/blog/?s=tableau&submit=Go

he American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, Erin M. Liang, Mark M. Holske, and Humberto Reynoso-Vallejo here, members of the research team evaluating the Health Care Cost Containment law (Chapter 224) from the Office of the State Auditor.

Using administrative data with a significant number of missing key variables (e.g. race/ethnicity) can be challenging in trying to answer specific evaluation questions. One of our research questions seeks to evaluate the impact of Chapter 224 on racial/ethnic disparities in health outcomes. Both administrative data sets available to us (the Massachusetts Medicaid Program (MassHealth) and the All Payer Claims Database (APCD)) have sparsely populated information about race and/or ethnicity. Since we were not able to apply imputation techniques due to the large number of missing values, we have to use alternative methods. As a proxy for race/ethnicity data, we used US Census Bureau Data and GIS mapping software.

Hot Tip: Use state level census data to account for missing racial/ethnic values. The US Census collects detailed race/ethnicity data at the state level every 10 years. Census data is publicly available on the American Fact Finder website. There are several data sets available, and this tool can aid in determining which data set best suits the needs of your project.

In this project, census data from 2010 will be applied to collected data spanning years 2006 to present, segmented by ZCTA (zip code tabulation area). Researchers should be aware that a zip code can be in more than one ZCTA. To learn about how zip codes are translated to ZCTAs, consult this census demonstration.

Large scale

This map was made using QGIS and census data. The map is segmented by cities/towns with the darker areas representing populations with a higher percentage of diverse residents.

Rad Resource:

  • The Census Bureau Help Line is 1-800-923-8282. A census representative can help to download the correct data.
  • In addition to QGIS, a large project with sufficient funding could utilize Tableau, an easy to use data analysis tool that allows users to visualize data with graphs, cross-tabs, and maps. While these functions can be done in other tools like Excel or ArcMap, Tableau allows a user to create a dashboard containing all visualizations on the same page.

The American Evaluation Association is celebrating Large Scale Evaluation Week. The contributions all this week to aea365 come from evaluators who have worked on the evaluation of the Health Care Cost Containment Law in Massachusetts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Greetings, fellow evaluation enthusiasts! My name is John Murphy and I am an Evaluation Associate on the Education Research and Measurement team at Cincinnati Children’s Hospital Medical Center. We provide evaluative support to various forms of learning and career development, ranging from clinical orientation to leadership and management training. One of our clients does extensive work surrounding quality improvement education. While we have provided them with evaluations of their courses, we have been fortunate enough to glean important tips and advice from their wealth of knowledge in measurement theory. One of the most basic tools used in improvement science is the annotated run chart, a simple form of the line chart. What makes the annotated run chart different from the typical line chart are annotations, small text snippets that show when an intervention or event has taken place.

Murphy

More and more, I have embraced annotations as being crucial to providing context and a story to a data representation. As an aficionado of data visualization, I have begun the quest for the perfect annotation. Here is what I have found so far:

Rad Resource: Stephanie Evergreen and Ann Emery provided an amazing resource within the virtual pages of this very blog! Their data visualization checklist not only reaffirmed my enthusiasm for the annotation, it also gave concrete guides for font size and text direction.

Rad Resource: What discussion about data visualization would be complete without mentioning Edward Tufte? The first chapter of his 2006 book Beautiful Evidence, entitled “Mapped Pictures”, discusses, in rich detail, the benefits of various techniques of providing context to images. Placing content in its proper space, scale, and time is crucial for making all genres of data representation tell a compelling story.

Hot Tip: If you are creating many annotated run charts that are updated frequently, consider investing in BI software such as Tableau. While Excel data labels are functional for one-shot data representations, more dynamic software can save time and provide more flexibility so annotations fit the story instead of being limited by the medium.

Good luck in telling your data stories!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top