Our names are Marc Wheeler and Salem Valentino and we are internal evaluators for Big Brothers Big Sisters of America.

There is a lot of buzz today about infographics.  Many of you may have thought about using infographics in your evaluation reports to try to translate your findings more effectively.  We recently took the plunge and incorporated infographics in our 2013 Youth Outcomes Report .

For example, we created the infographic below to clarify the theoretical connection between the short-term outcomes we currently measure for each youth and those long-term outcomes of interest to many stakeholders. The graphic summarized a large quantity of research literature in a single infographic that was easily interpretable and concise.

Lessons Learned:

1. High-quality infographics require a certain level of expertise; as we didn’t have the relevant experience in house, we contracted with an external graphic designer who delivered great results.
2. Beyond experience, it also takes time to develop your infographics and get them right.  First, you need to figure out the story behind your data.  Then, what are the best ways to illustrate this story, while remaining true to the data?  For instance, we wanted to communicate effect sizes but didn’t want to take up space in our report explaining what they meant to a lay audience.  So instead, we developed the visualization below to better illustrate our story of the magnitude of youth outcomes.  Lastly, infographics require a number of iterations and can benefit from the input of diverse audiences.  Budget your time accordingly.

1. Understanding your project needs will help you choose the right designer. We looked at several resources on the internet to find a graphic designer. For us it was important that our work with the designer was collaborative; we wanted to ensure the quality of the evaluation content. Due to our timeline, it was also important that our designer could design the entire report and not merely the infographics.  You may also want to ask your designer how comfortable they are with Excel or other types of data you will be using in the report.

Rad Resources:

Elissa Schloesser at Visual Voice – our designer’s 5 Steps for Translating Evaluation Findings into Infographics

Visual.ly’s Marketplace service will find a designer for you and help you create an infographic for one price.

Easel.ly is a website where you can create your own infographic for free.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization · evaluation reporting · infographics

Greetings! I’m Nichole Stewart, a doctoral student in UMBC’s Public Policy program in the evaluation and analytical methods track. I currently work as an analyst, data manager, and evaluator across a few different sites including Baltimore Integration Partnership, Baltimore Workforce Funders Collaborative, and Carson Research Consulting Inc.

Lessons Learned: The Growing Role of Data Science for the “Little” Data in Program Evaluation. Evaluators are increasingly engaged in data science along every step of the evaluation cycle. Collecting participant-level data and developing indicators to measure program outputs and outcomes is now only a small part of the puzzle. Evaluators are working with more complex data sources (administrative data), navigating and querying data management systems (ETO), exploring advanced analytic methods (propensity score matching), and using technology to visualize evaluation findings (R, Tableau).

Evaluators Also Use Big Data.  Large secondary datasets are appropriate in needs assessments and for measuring population-level outcomes. Community-level data, or data available for small levels of geography, provide context and can be used to derive neighborhood indicators. Evaluators must be able to not only access and manipulate this and other kinds of Big Data but to ultimately learn to use data science to maximize the value of the data.

Rad Resource: The American Community Survey (ACS)  is an especially rich, although recently controversial, Big Data resource for evaluators. The survey offers a wide range of data elements for areas as small as the census block and as specific as the percent of carpoolers working in service occupations in a census tract.

Hot Tips:

• View the latest technical document for the ACS 5-Year estimates for a list of available data elements (p. 43).
• Visit the National Historical Geographic Information System to download Census and ACS data as well as GIS boundary files for mapping.

Rad Resource: The Census Bureau’s OnTheMap application is an interactive web-based tool that provides counts of jobs and workers and information about commuting patterns that I explored in an AEA Coffee Break webinar.

Lessons Learned: Data Science is Storytelling: Below is a map of unemployment rates by census tract from the ACS for Baltimore City and surrounding counties.  This unemployment data is overlaid with data extracted from OntheMap depicting job density and the top 25 work destinations for Baltimore City residents.  The map shows that 1) there are high concentrations of unemployed residents in inner-city Baltimore compared to other areas, 2) jobs in the region are concentrated in Downtown Baltimore and along public transportation lines and the beltway, and 3) many Baltimore City workers commute to areas in the surrounding counties for work.  Alone, these two datasets are robust but their power lies in visualizing data and interpreting relevant intersections between them.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

big data · community mapping · Data · GIS · mapping

I’m Taj Carson, the President of Carson Research Consulting (CRC) in Baltimore, MD. CRC is a research and evaluation consulting firm and we’ve seen first-hand how neighborhood-level data is increasingly being used for tasks such as identifying community conditions and trends or measuring population-level outcomes in research and evaluation.

Recently, I was in a meeting where a group of human service providers met to discuss where to locate a program for pregnant and parenting teen mothers in Baltimore City. They realized that what they really needed to know was— which communities have the highest teen birth rates in Baltimore?  A colleague went into a storage room and retrieved a huge, styrofoam-backed map from 2008, showing teen birth rates across the city, so the group could decide where to locate the program. While I was impressed with the fact that they valued data enough to use it in the early planning stages, and that they were actually able to remember where they had stashed a map from 2008, I was encouraged to get the word out about the DataMind as an interactive mapping tool that would allow them to see this information without rummaging around in a closet.

Lessons Learned:

• Creative, forward thinking program planners know when to bring data to the table to make decisions.
• While there is a wealth of available data on our communities, we are short on ways to visualize that information on the spot and identify patterns across different parts of the city.
• Visualizing several data sources together in an interactive map allows for a more complex understanding of data, even if it is on paper.

Rad Resource: The Baltimore DataMind is an interactive mapping tool that allows users to visualize data for Baltimore City neighborhoods to promote collaboration, advocacy, informed decisions, and effective policy making. Users can compare data across neighborhoods, create a community profile of a neighborhood, and combine data indicators and community resources and assets in one map. In the above-described situation, the Baltimore DataMind could have provided the information this group needed by looking at the “Teen Birth Rate” data in the Children and Family Health Indicators section of the map. They also could have mapped out the location of schools and community health centers in those neighborhoods. Hot Tips:

• You Don’t Have to be a GIS Expert: Neighborhood-level data could have easily been pulled up in theBaltimore Datamind “widget” in an easy-to-use interface developed by Policy Map.
• Data can be shared: The maps can be printed and then shared with neighborhood stakeholders, funders and community residents.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

community mapping · GIS · neighborhood mapping

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor. Earlier this month, you heard from Soledad Muñiz about  InsightShare’s work with participatory video (PV) for monitoring and evaluation. PV isn’t the only area in which InsightShare is leading the way.

Hot Tip – Photostories: Photostories are a “new format developed by InsightShare for sharing project case studies in an accessible, fun and visually engaging format.” Each photostory is basically a comic that documents a project’s methodology. Here is an example. Click on it to open the high resolution version (it can take a moment to download).

Rad Resource – Example Photostories: More examples from InsightShare may be found online here.

Lessons Learned Using Photostories: I spoke with Soledad via Skype to ask her about InsightShare’s use of photostories. She noted that they offered an engaging option for telling what otherwise can be a dry subject, research methods. They could help to raise the interest of stakeholders and could span linguistic barriers, whether those be due to limited literacy or different languages spoken among stakeholders. Soledad noted that they used the Comic Life software to create their photostories and I decided to give it a try.

Rad Resource – Comic Life: Comic Life is software that helps you to quickly make comics from photos that you have on hand. It is extremely easy to use, with a drag and drop interface and multiple templates. When I demonstrated this in a workshop, an attendee asked if you couldn’t just create comics in another program, perhaps Microsoft’s Publisher or even PowerPoint. Yes! But, Comic Life makes the process considerably easier. The software indexed the files on my hard drive and then I could drag in those I wished to use, add text, and with a couple of clicks, everything was aligned and beautifully laid out and ready to publish. My mind began swimming with the possibilities for using Comic Life in different contexts. After the trial period ended, I happily paid the $29 for the version for my Mac laptop. It is also available for windows, and as a $4.99 ap for iOS devices.

Get Involved – Share your innovative evaluation communication ideas and win an eStudy workshop registration: I’ve written before, encouraging you to think imaginatively about engaging stakeholders around evaluation communication and reporting. Any time before May 10, share your idea for ‘out of the box’ options for evaluation communicating and reporting via the comments on the post. We’ll randomly select one contributor from among all of those making suggestions and you can win a free registration to Kylie Hutchinson’s upcoming eStudy “An Executive Summary is Not Enough: Effective Reporting Techniques for Evaluators” or any AEA eStudy of your choice offered between now and December 31, 2013. New eStudies are added monthly and the current list is online here. While you are welcome to comment multiple times, we’ll only consider one entry per person. You do not need to be an AEA member to enter.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings. I’m Paul Barese, Owner-Director of Quimera, a small business based in Washington, DC. Quimera isn’t a traditional media house, rather, teams are interdisciplinary and our approaches are informed by social science and qualitative research, change management, film studies, and social enterprise.

Quimera helps clients identify innovative ways to use video. In some projects video-based approaches are integrated into the data-collection and analysis process as part of the evaluation toolkit (ie: participatory and ethnographic methods– video as process). In other projects, video documents data collection and/or discussion of findings for reporting and dissemination–video as product). In some cases it’s both. We’ve co-developed projects as part of strategic design, evaluation, knowledge management/ training, reporting/ dissemination/ communications/public relations.

Quimera focuses on the added-value of video, from conceptualization, co-design, production, through editing and dissemination. At times Quimera is a member of the research team, using video creation as part of the data collection process, and raw footage as qualitative data contributing to analysis. Quimera’s work is collaborative, ethnographic where appropriate (applied) and it’s common that stakeholders, beneficiaries and research/ evaluation members are involved in content creation. Quimera leads an editing process that includes varying degrees of review and collaboration with the client and little if any participation from beneficiaries.

Here’s an example of a video report that we made (if this does not display in your email or browser, click through directly to: http://www.youtube.com/watch?v=XxgTw2Ya5DQ)

Lessons Learned: When working with video it can be important for clients to think about organizational dynamics, politics, and the relationships between functional areas.

• Look at maximizing (and co-funding) investments in production and editing by coordinating with other departments. Folks doing knowledge management, training, and communications are often interested in video and could be included in the planning process prior to field production so crews capture that extra bit of content which makes the raw footage useful to colleagues. Multi-purposing video can be important for financial and collaborative purposes.

• Discussions with colleagues in other departments, ie: communications, can help defuse what might become tense or frustrating relationships between uses and dissemination of video created by “evaluation” but that carry interest, relevance or transparency concerns to communications.

• Integration of video into evaluation can impact functional roles within an existing organization, required skill sets, work processes, equipment and IT infrastructure. It’s useful to look at bigger picture issues and longer-term ramifications.

• Participatory approaches to content creation (putting a camera into the hands of a beneficiary) are valuable on many levels… engagement, dynamics, transparency, accountability, comfort, openness, sensitivity, data-quality.

Lessons Learned – Taming Technology: Don’t overlook audio equipment (external microphones). Audiences are forgiving when it comes to rough visuals but bad audio is the surest way to lose viewer attention and almost guarantee the click off.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor. On occasion, I offer training on different aspects of evaluation. Following a recent workshop at the Minnesota Evaluation Studies Institute, I posted the following to AEA’s LinkedIn Group:

Do you know of songs with an evaluation bent? This might be filed under ‘weird question of the week.’ I am teaching workshops on evaluation, in particular on data visualization. I regularly use two songs as transitions- “Do you want the truth or something beautiful” from Paloma Faith (a great bluesy rumination) and “I Love Charts” from Sid the Sience Kid (a children’s song that shouts “Charts rule!”).

Ever-responsive, the LinkedIn group came back with a plethora of suggestions.

Rad Resources – Evaluation Related Songs: Suggestions from our colleagues included:

• Paul Duignan suggested a compilation of 20 songs related to No Child Left Behind
• Keren Brooks added What If No One’s Watching, Fire Door, and My IQ from Ani DiFranco
• Alan Gumbel went back to the 1973 musical Lost Horizons to consider The World Is a Circle and Question Me an Answer
• Seanna Hasson suggested What’s the Use from Jamie Lidell
• Colleen Duggan recommended Cognitive Bias from Brad Wray
• And both Colleen and Nayiri Dolabjian suggested Terry Smutylo’s The Impact Blues
• Allison Titcomb’s recommendation ranged widely from My Way by Paul Anka (popularized by Frank Sinatra and others), to STAT-60 from MC Lars, and the Indiana RISE Evaluation Song from K Downey
• Elisa Avila recommended a song I love and hadn’t thought of – Effect and Cause from the White Strips
• Cliff Carr added (I can’t get no) Satisfaction from the Rolling Stones
• Tongue in cheek, Harry Jones suggested ‘the ultimate negative assessmet,’ God May Forgive You, But I Won’t

Hot Tip – AEA on LinkedIn: AEA’s LinkedIn group is a great start to connect with colleagues far and wide for guidance on questions from the lighthearted to the fundamental. Members and nonmembers alike are welcome to join the conversation.

Get Involved – Suggest a Song: Any songs we’re still missing? Share your ideas via the comments. Thanks!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I am Manny Straehle and I work with International Credentialing Associates (ICA). ICA is a measurement consulting firm with expertise in developing certificates, certification, and licensing programs. We are committed to producing reports and visualizations that are simple and easy to understand for the clients we serve, but we need your help.

Hot Tip – Enter our Data Visualization Competition: With the sponsorship of AEA’s Data Visualization and Reporting TIG, we are launching a data visualization competition. The winner will receive $500.

ICA often generates data reports for our clients – and revising the look and ease of this data report is the basis for this data visualization competition.

Don’t worry if you aren’t familiar with certification and licensing programs, or if some of the terms below are new to you. The contest involves improved organization and display – no data analysis required.  The submissions will be reviewed by our panel of judges, including me, Rory McCorkle, DVR TIG founder Stephanie Evergreen, DVR TIG Co-chair Johanna Morariu, and other experts.

Task:       To create a data visualization-centered report, using a provided dummy set of data.

Hot Tip – Include These Required Report Components:

• ICA’s name, logo, and color schema (available from the ICA website)
• Space for customer name
• The following data points, from the provided Excel spreadsheet:
  • Key (correct answer)
  • P+ (item difficulty)
    • Percent of candidates who answer the question correctly
  • Distractor Analysis (number and percentage) across three performance levels
  • Rbis (item discrimination)
    • Correlation between performance on an item and total performance
  • Recommended actions

Hot Tip – Attend to the Evaluation Criteria:

• Uses basic spreadsheet software (Excel, Open Office, etc.) for data visualizations
• Report fits on one page
• Required report components are present
• Text is legible
• Data visualizations are easily interpreted
• Transfer of graphs from spreadsheet to report is easy
• Report uses best practices in data visualization design
  • Layout and use of space
  • Use of color and contrast
  • Inclusion of appropriate text
  • Use of data visualizations appropriate to data
  • Contribution of design to overall comprehension of information

Hot Tip – Follow Our Submission Instructions:

• Download the Excel spreadsheet with a data set you can use (please note the file is dummy data).
• Submit your revision, both report in PDF format and spreadsheet file, to info@intlcred.com by 15 April 2013.
• The winner will be announced on 15 May 2013 and contacted by email.
• All entries become the property of ICA upon submission.  By submitting an entry, all Entrants agree to the Official Rules of Entry.
• Questions can be directed to info@intlcred.com

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization · reports

My name is Susan Kistler and I am the Executive Director of the American Evaluation Association and the regularly Saturday contributor on area365. Next week I am traveling to Minnesota to give a workshop at the Minnesota Evaluation Studies Institute on “Data Visualization, Presentation, and Reporting That Engages Stakeholders, Elevates Your Message, and Enhances Use.”

One component of our workshop will focus on the use of video in evaluation, hence last week’s aea365 post about InsightShare’s use of Participatory Video. I also had at hand examples of reporting, using comics and humor (I’ll write a bit more about these in a future post). What I couldn’t find was an example of leveraging humor and video for evaluation reporting, or even a reporting video that could be described as lighthearted. So I had one made.

Lessons Learned – How to Get a Video Made for $5: I wrote out a short script expressing three key findings from a survey of the services in an imaginary animal hospital. And, who better to speak for the animals than an animal himself if Dr. Doolittle isn’t around?

So, I turned to Fiverr. I have written about Fiverr before – they are a website where people do tasks for $5. “ThePianist1” has an offer where he makes a cute green frog talk for $5 and say whatever you’d like. This was exactly what I needed.

$5 and two days later, I had my video. (If you are receiving this post via email, and the video doesn’t appear below, please click through back to the site – the example is worth a look.)

Hot Tip – Be Vigilant and Use Your Imagination: In February of 2012, John Gargani wrote on EvalBlog that he would be celebrating EVALentine’s day on February 15. Not only was he celebrating it, he had made EVALentine’s cards and real, usable, EVAL postage stamps. I got my very own sheet and did my happy dance. John is one of those people who come up with ideas that you wish you had thought of, but (at least in my case) rarely do. John helps me to see the world in a different light, full of opportunity. When I came across the frog a few months ago, I was ready to think about whether there might be a role for him in my own work.

Get Involved: What is the most innovative evaluation reporting technique that you have seen used? Was it effective? Any ideas that you have considered but haven’t yet had the opportunity to try?

No tags

Hello all, I’m Kim Firth Leonard, American Evaluation Association (AEA) member since 2008, and President of local affiliate OPEN, the Oregon Program Evaluators Network. I currently work at Marylhurst University in Portland, Oregon, primarily on assessment of student learning and academic programs, though I also dabble in institutional research. I also do contract work in program evaluation via Leonard Research and Evaluation LLC.

Rad Resource – actionable data: I started the blog actionable data in 2011 and post somewhat regularly (a few times per month whenever possible) with a handful of friends and co-authors. The blog advocates for the collection of meaningful and useful data, and for wise use of that data. Our posts span a range of topics often related to program evaluation, though most focus more specifically on data and data use.

Hot Tips – favorite posts: Here are a few mostly recent, favorite and/or most visited posts authored by yours truly so far:

• 11/29/2011 – What data is not actionable? An attempt to further define actionable data is a post in which I babble about what I think actionable data is (and is not) and pose several questions that I myself don’t have answers to.
• 11/23/2012 – I posted on My love/hate relationship with pie charts. Though my opinion is already shifting on this topic, I love the conversation this post generated.
• 12/31/2012 – The Functional Art and growing as an actionable data advocate is the first in what I hope will be a series of posts about my experience in Alberto Cairo’s Infographics and Data Visualization MOOC (massive open online course) and includes a New Year’s resolution declaration.
• 1/11/2013 – Most recently, I posted Analysis Paralysis discussing decision fatigue, noting that “if we want decision makers to use data, we should probably understand decision fatigue.”

(Share Clip)

Lessons Learned – why I blog: For me, blogging is an opportunity to question, explore, and learn as well as to share what I know. To think together with my co-authors and anyone willing to read (and comment) along with us! A ‘manifesto’ for actionable data is here.

I also ‘micro blog’ on Twitter (@KimFLeonard), which has been a wonderful way to engage others with my blog and to find people who are doing interesting work. Between the blog and Twitter, I have discovered many wonderful resources and connected to other great evaluators (including Sheila B. Robinson, who is graciously co-authoring a series of posts with me).

Lessons Learned – what I’ve learned: How liberating and enlightening it can be to throw an idea online. Or to ponder something ‘out loud.’ And that blogging, especially when accompanied by conversation via social media, can be an amazing networking and learning tool.

This winter, we’re continuing our series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Data · data visualization · twitter

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director and aea365’s regular Saturday contributor. Except, today is Sunday, because Friday and Saturday brought snowfall for the record books and here on the Southcoast that snow came with 40 mile per hour sustained winds and gusts far exceeding that. Power outage in our area and neighboring towns exceeds 90%. I am blessed to live on a campus with backup generators providing heat in some buildings and the capacity to serve warm meals in the dining hall, even as I sit typing this via candlelight.

We’re all glued to charged mobile devices and the our electrical provider’s online outage map with near real-time updates on customers affected by the storm. See the swath of black at the shoulder of the flexed arm that is Cape Cod? That’s where we live and work, in the section marked 91%-100% affected. A lack of electricity provides plenty of time to reflect on lessons learned beyond ‘keep plenty of spare batteries on hand’.

NSTAR Outage Map

Lesson Learned for Reporting:

• For those who are information starved, any information is seductive. People are trying to make decisions from the relatively mundane (do I eat the ice cream before it melts) to the fundamental (do I move to a shelter today because it is 40 degrees and dropping in my house and it is difficult to keep the kids warm).
• Not knowing what came before, limits data quality for decision-making. The outage map provides a snapshot in time, but no history. We gather round, trying to remember if the next town over has changed color from the last time we looked as a hint as to whether the repair trucks are headed this way.
• Aggregating data illustrates trends but may provide little guidance for those seeking to make decisions on the individual rather than the systemic level. Knowing that your town will likely have full power restored by Thursday is of little use in understanding when power is coming to your street and your home.
• Information is of little value without context. Like in social programs, we’ve learned that in some cases, one major fix can improve prospects for thousands of families and yet for others it can take multiple line repairs to work down a single street to the house at the end.

Lesson Learned for Life: I am thankful to everyone, from the linemen working to restore power to the kids who are helping to dig things out to the townspeople asking one another ‘how can I help’ and ‘do you need a hand.’ All is well in the grand scheme of things. We’re safe, sound, and gaining a newfound appreciation for the art of conversation.

[update, today is actually Tuesday – I thought that I had gotten this posted on Sunday, but the adventures of posting from one’s iphone are not to be understated, so this is backdated! The electricity has returned and we’re digging out.]

No tags