AEA365 | A Tip-a-Day by and for Evaluators

TAG | Tableau

We are Ana Flores and Joshua Paul at Volunteers of America – Los Angeles. At the AEA Evaluation 2016 conference, we presented a panel entitled “A Picture is Worth a Thousand Words…But Will They Use It?”. Today, we want to provide additional information regarding how to make data more user-friendly.

The Evaluation Department at Volunteers of America – Los Angeles (VOALA) is tasked with providing evaluation services as needed to more than 70 social service programs. Staff in these programs are dedicated to helping people and many find data unappealing. Addressing communication barriers has given us the opportunity to learn a number of lessons.

Lessons Learned #1: Needs Change, Open a Dialog with Succinct Visualizations  

Understanding stakeholder needs and how they fit into a program model is a major part of any evaluator’s task. Unfortunately, we have found that stakeholder needs and program models can change rapidly, and stakeholders do not always volunteer information about these changes.

We were once mystified as to why one of our programs — whose initial purpose was to connect with and refer homeless veterans to local services — had such poor monitoring results. Traditional reporting methods failed to open a dialog that could bring the core problem to light. After months of discussion, we tried a new visualization-based design (see image) that demonstrated the discrepancy between the goal and present performance and prompted program leadership to identify the issue. The staff had been focused on the transportation of clients to appointments, a secondary program activity, which had not originally been designated as important to track.

internal-eval-image-1

internal-eval-2

Lessons Learned #2: Only Show What You Need to Show

 

Past reports for many of our programs provided detailed data, presenting every single outcome for individual clients. However, this level of information was not necessary for program performance discussions and was a distraction from the overall outcomes included in the report. Using Tableau, the detailed information was removed, and only overall outcome percentages and targets were kept on the graph. With outcomes presented this way, VOALA upper management was able to get the information they needed to make program recommendations and help program directors implement better practices.

 

Lessons Learned #3: Use Interactivity

Giving your audience an opportunity to control the data makes it easier for them to make inferences about the information. Visual analysis programs, like Tableau, allow us to provide interactive reports so that upper management and program directors can filter results by key demographics or periods of time, depending on what is useful to them.

 

Having these types of “quick snapshot” visualizations has helped upper management at VOALA communicate recommendations with programs.

Experimenting with these different data visualization techniques has improved our discussions with key staff, helping us ask hard questions while reducing staff resistance to data. Otherwise, the response to “Why is this benchmark never reached?” might just be silence.

The American Evaluation Association is celebrating Internal Evaluation (IE) Topical Interest Group Week. The contributions all this week to aea365 come from our IE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Nina Potter and I am the Director of Assessment for the College of Education at San Diego State University.  As the Director of Assessment, a large part of my job entails working with faculty and administrators in implementing program assessments plans from start to finish – designing assessments to measure program outcomes, collecting the assessment results electronically and then using the results to inform instruction and program design.  I thought that in a College of Education this was going to be easy.  I thought that everyone would understand the importance of collecting assessment data and sharing the data with colleagues in the College.   The reality is that there is a range of knowledge and ability as well as in the willingness to share data with colleagues.

Rad Resource: Sometimes what appears to be a lack of willingness to share data is really lack of time.  Faculty are busy teaching courses and doing their own research, sometimes it can be hard to find time to get a large group together in order to review data.  A tool like Tableau Server allows us to share data so faculty and administrators can review it from anywhere.  With Tableau you can link directly to data sources and schedule regular times to refresh the data so everyone can access the most up to date information easily.  During face-to-face meetings, we can spend more time focused on what to do about the assessment results rather than on summarizing the results.

Hot Tip: Keep the charts as simple as possible so they are easy to understand at a glance.  A chart like the one below (NOT actual student data) can give a quick picture comparing how students are performing on different assignments designed to measure the same standards or learning outcomes.  Since people can access the charts at any time, I won’t always be around to answer questions.

Untitled

 

Lessons Learned: Before sharing data at the course level, faculty have to have trusting relationships with each other.  There are a variety of reasons why some faculty may not be willing to share the results from their courses.  Examples include individual faculty being insecure about their teaching ability or faculty feeling competitive with one another.  I usually start be sharing data aggregated in such a way that results by individual faculty are not visible until I have developed that trust.

The American Evaluation Association is celebrating Assessment in Higher Education (AHE) TIG Week. The contributions all this week to aea365 come from AHE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Julie Lo and I’m a research assistant at Public Profit.  We help youth service agencies and educational organizations manage what matters.  I want to share with you a way of presenting survey data through an embedded interactive interface created using the program Tableau Desktop and later shared online using Tableau Public.

We process thousands of stakeholder surveys each year for our clients and were thrilled when one of them approached us about creating a method of publishing a city-wide survey of thousands of high school youth to a wider audience.  We found a way to consolidate what in past years has been a 250-page report to an interactive set of data visualizations:

Emphasizing the many different things to consider when creating any kind of data visualization, I’ve included some practices that we will be sure to consider for future projects like this.

Lesson Learned: Which came first: function or design? 

The first leg of the project involves building out of the basic frame: pages, buttons, selecting the best viz to tell the story. Leave enough fuel in your project’s gas tank to be able to do this. Share it with your client for feedback to make sure that it not only works well, but is also designed in a way that is accessible for youth, teachers, and policy analysts alike!

We worked through several drafts before arriving at the final product both internally and with our client.  This helped us to make decisions around what types of visualizations work best for the data on hand.  Luckily, our client has been working on this project for multiple years and was able to share valuable insights on what information to emphasize for the audience and the “look” that would be most appealing.

Lesson Learned: How fit is your data?

Tableau Public has excellent built-in online sharing features and is seamlessly linked to Tableau Desktop making the work of transferring your work online quick and easy. However, Tableau Public does not have a way to suppress the underlying data, posing potential issues around the confidentiality of individual survey takers.  We got around this by using summarized data that we prepared outside of Tableau in SPSS and used summarized data for the underlying data for the Tableau Public file.  This is an important step to consider when planning your project timeline since it adds a little bit of data-prep work.

Lesson Learned: So many questions, where to begin? 

We worked closely with our client to help develop themes to group survey items together and to select ways to filter data by demographics. This creates further utility for both our clients and those they serve. Presenting questions grouped together by theme can help viewers navigate survey results for their populations of interest, helps to make sense of data and keeps pages free of the dreaded data-viz clutter.

Rad Resources – Related aea365 posts:

Rad Resources:

We’re celebrating Data Visualization and Reporting Week with our colleagues in the DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

·

I’m Susan Kistler, the American Evaluation Association’s Executive Director. I contribute each Saturday’s post to aea365. I’m striving to improve my capacity to use Tableau, a data exploration and visualization tool. I’ve read the manual, watched their video tutorials, will be attending training, and have spent time working with the product on my own. However, as with any complex tool, the devil is in the details and I occasionally find myself caught up with how to make it do exactly what I want as efficiently and effectively as possible.

Today, I wanted to share three tools that I use for support as I strive to use new software and hardware, and to get the most from products with which I have worked for years.

Rad Resource – Product Discussion Lists: Discussion lists, or forums, devoted to a particular product are usually populated by both users and representatives from the manufacturer. They are places to learn from others who have ‘been there, done that.’ A quick check of Tableau’s site and I found Support -> Forums. Support forums tend to be fairly no-nonsense question-and-answer spaces.

Rad Resource – Experts Exchange: Experts Exchange is a site where you can post tech questions on almost any topic or platform and get answers from thousands of other users. It is like having ready access to hundreds of discussion lists covering multiple products. Experts Exchange isn’t free, but for $100 per year, I’ve received multiple specific answers tailored to my problem. I’ve found those who respond to be very sophisticated users who know the details of a particular product.

Rad Resource – Users Groups: Sometimes, you want a bit more than Q&A. You may be seeking demonstrations, meetups, peer-critiques, examples, and networking. Users Groups bring together those with an interest in a particular product and usually supplement what you’d find on a discussion list with other streams for capacity building. User Groups used to be location-dependent, so there were often many groups for a particular product, but increasingly there are virtual groups as well. Apple and SAS have both set up spaces on their website to support users groups for their products. And, if you know of other users groups that may be of interest to evaluators, please share them in the discussion section for this post.

Hot Tip – Join the Tableau Virtual Users Group: I like users groups. They combine product support and demonstration with networking and commiseration. I’m starting a Tableau virtual users group and welcome anyone working in evaluation or a related field who’d like to join us to add to the discussion on AEA’s LinkedIn group here. We’re hoping to meet monthly via webinar with short demos, peer-critique, time for questions and networking, etc.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Archives

To top