AEA365 | A Tip-a-Day by and for Evaluators

TAG | GIS

Hi! Our names are Carrie Wiley and Matt Reeder and we are Senior Research Scientists at the Human Resources Research Organization (HumRRO). We would like to share an abbreviated version of our demonstration session presented at the 2016 annual meeting in Atlanta on how to create map data in R. It sounds like a daunting task, but it is far easier than it seems.

In addition to the many tools and resources that exist to help guide evaluators to create more effective tables and graphs, geographic mapping could also be a great benefit to identify and demonstrate geographical patterns. The use of Geographic Information System (GIS) mapping as an effective evaluation tool might be perceived by many as a rather intimidating technique, since most evaluators are not formally trained in GIS. In our work, we often deal with naturally occurring large-scale data (e.g., state-level data, school districts, counties, ZIP codes) that can be displayed in more effective ways than a traditional table. Drawing maps really just requires coordinates, and for very basic maps, R provides those coordinates in a nicely formatted file.

Hot Tips:

All you need to get started is:

GIS Basics:

In order to map data, you need to draw boundaries. Those boundary data are in shapefiles (.shp) which contain latitude and longitude coordinates of the boundaries you want to draw. The Census Bureau TIGER files (Topologically Integrated Geographic Encoding and Referencing) make various cartographic boundary shapefiles available for download, or you can use built-in R packages that essentially pull the data for you.

Mapping the Data:

Our example plots a heatmap of the number of craft breweries in each state.

  1. Retrieve the publicly available craft brewery directory: https://www.brewersassociation.org/directories/breweries/

2. Install the following R packages:

 a. library(dplyr)

 b. library(ggplot2)

 c. library(mapproj)

3. Data excerpt:

4. Load the boundary data from maps() (a ggplot() dependency):

 a. states <- map_data(“state”)

 b. Data excerpt:

5. Get counts of breweries by state and merge with the coordinates file:

6. Plot the heatmap:

So, based on this map, if you are an avid fan of craft beer, California, Washington, and Colorado are good places to check out. Of course, these are raw counts—creating a heatmap that accounts for population density would be more useful. If you are a coffee drinker, find a publicly available coffee shop database and practice your new skills plotting a heatmap of coffee shops! 

Rad Resources:

Using different combinations of R packages and Census data, you can make heatmaps by county, and school districts, and bubble charts by ZIP code.

Useful Census data:

Useful R packages

  • library(zipcode)
  • library(maps)

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hello. My name is Chris Michael Kirk and I am the Director of Mission Development at Atlantic Health System in New Jersey where I lead our Center for Population Health Sciences. At the Center, we conduct applied population health research and disseminate and evaluate community health interventions, including our Healthy Communities Initiative designed to reducing health disparities in our region.

Our team (of mostly nurses and health educators) was doing amazing work in our communities, but struggled to prioritize these efforts and evaluate their effectiveness. While on a conference call for the health disparities workgroup, one member offered that we were providing an intervention targeted for low-income, Spanish-speaking populations in a wealthy suburb nearby. After I picked myself up off the floor, I asked the obvious question: “Why are we dedicating our limited resources to address health disparities where they do not exist?” Unfortunately, the answer was: “Because they asked us to”.

This exemplified the need to move our team toward data-based decision-making to guide program dissemination and evaluation metrics, but to do so in an accessible manner that built upon existing strengths and history.

Hot Tip: To move beyond jargon, we utilized GIS mapping of our community on key socioeconomic and demographic indicators alongside hospital utilization data to open up a discussion that helped our staff move from defensiveness over not responding to community requests to prioritization toward areas of greater need. By presenting maps accompanied by data tables, we matched the “hard data” with their lived experience working in the laundromats, grocery stores, and Hindu temples in those areas. “I know that neighborhood” was the rallying cry, creating space to build on their expertise and identify intervention opportunities in target communities.  

Lessons Learned: Our staff initially reacted to the word “evaluation” as if it were Ebola. They expressed unexpected confusion and concern and felt threatened. We discovered that it was the semantics, not the practice, that put up a barrier. On their own, staff developed elaborate systems for tracking repeat participants in community-based events. They wanted to know if their efforts were making an impact. By building on this desire and re-framing the question, we engaged them in a process to redesign the Initiative, eventually adding place-based approaches to our direct outreach.

Rad Resource: The Vulnerable Populations Footprint from Community Commons was one tool that we used to quickly visualize our communities and guide the conversation. By giving staff the ability to “play around” with the map, you can open up new lines of inquiry and give them buy-in on the resulting decisions.

The American Evaluation Association is celebrating CP TIG Week with our colleagues in the Community Psychology Topical Interest Group. The contributions all week come from CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Lisle Hites, Chair of the Needs Assessment TIG and Director of the Evaluation and Assessment Unit (EAU) at the University of Alabama at Birmingham (UAB). Today’s posting is about the use of data visualization to enhance your needs assessment.

Recently, my team worked with a state agency to help them identify potential sites for a pre-k development initiative. We used ArcGIS 10.2 Geographic Information Systems (GIS) technology to geocode and map all child care centers and grant applicants within the state. In turn, these data were displayed on an interactive, web-based map using ESRI’s ArcOnline platform. Supplemental data regarding percentage of people in poverty were added to the map to enhance the decision making process for policy makers (American Community Survey Census).

Displaying these multiple sets of data visually allowed state representatives to see the highest concentrations of four year olds in the state as well as potential gaps in service coverage by existing pre-k programs. In other words, these data were used to reduce the potential for duplication of services and to identify areas of greatest need.

Lessons Learned:

  1. Needs assessments can be conducted in a variety of ways using existing data in new and innovative ways.
  2. While state representatives had ideas of what they wanted to know, data visualization led them to refine their questions and identify additional sources of information to support their “data-driven” decision.
  3. Hardcopy paper maps of each county did not provide enough geographic detail of childcare facilities. To maximize the large amount of disparate data, an online interactive mapping platform was critical to the success of this project.

Rad Resources:

ArcGIS Online (n.d.). The mapping platform for your organization. ESRI.

ArcNews (2013, Summer). ArcGIS 10.2 brings transformational capabilities to users. ESRI.

Azzam, T., & Robinson, D. (2013). GIS in evaluation: Utilizing the power of geographic

information systems to represent evaluation data. American Journal of Evaluation, 34(2),

207-224. doi: 10.1177/1098214012461710

Evergreen, S. (2013). Presenting data effectively: Community your findings for maximum

impact. NY: Sage Publications.

United States Census Bureau (2015). American community census.

The American Evaluation Association is celebrating Needs Assessment (NA) TIG Week with our colleagues in the Needs Assessment Topical Interest Group. The contributions all this week to aea365 come from our NA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hello, Erin M. Liang, Mark M. Holske, and Humberto Reynoso-Vallejo here, members of the research team evaluating the Health Care Cost Containment law (Chapter 224) from the Office of the State Auditor.

Using administrative data with a significant number of missing key variables (e.g. race/ethnicity) can be challenging in trying to answer specific evaluation questions. One of our research questions seeks to evaluate the impact of Chapter 224 on racial/ethnic disparities in health outcomes. Both administrative data sets available to us (the Massachusetts Medicaid Program (MassHealth) and the All Payer Claims Database (APCD)) have sparsely populated information about race and/or ethnicity. Since we were not able to apply imputation techniques due to the large number of missing values, we have to use alternative methods. As a proxy for race/ethnicity data, we used US Census Bureau Data and GIS mapping software.

Hot Tip: Use state level census data to account for missing racial/ethnic values. The US Census collects detailed race/ethnicity data at the state level every 10 years. Census data is publicly available on the American Fact Finder website. There are several data sets available, and this tool can aid in determining which data set best suits the needs of your project.

In this project, census data from 2010 will be applied to collected data spanning years 2006 to present, segmented by ZCTA (zip code tabulation area). Researchers should be aware that a zip code can be in more than one ZCTA. To learn about how zip codes are translated to ZCTAs, consult this census demonstration.

Large scale

This map was made using QGIS and census data. The map is segmented by cities/towns with the darker areas representing populations with a higher percentage of diverse residents.

Rad Resource:

  • The Census Bureau Help Line is 1-800-923-8282. A census representative can help to download the correct data.
  • In addition to QGIS, a large project with sufficient funding could utilize Tableau, an easy to use data analysis tool that allows users to visualize data with graphs, cross-tabs, and maps. While these functions can be done in other tools like Excel or ArcMap, Tableau allows a user to create a dashboard containing all visualizations on the same page.

The American Evaluation Association is celebrating Large Scale Evaluation Week. The contributions all this week to aea365 come from evaluators who have worked on the evaluation of the Health Care Cost Containment Law in Massachusetts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Greetings! I’m Nichole Stewart, a doctoral student in UMBC’s Public Policy program in the evaluation and analytical methods track. I currently work as an analyst, data manager, and evaluator across a few different sites including Baltimore Integration Partnership, Baltimore Workforce Funders Collaborative, and Carson Research Consulting Inc.

Lessons Learned: The Growing Role of Data Science for the “Little” Data in Program Evaluation. Evaluators are increasingly engaged in data science along every step of the evaluation cycle. Collecting participant-level data and developing indicators to measure program outputs and outcomes is now only a small part of the puzzle. Evaluators are working with more complex data sources (administrative data), navigating and querying data management systems (ETO), exploring advanced analytic methods (propensity score matching), and using technology to visualize evaluation findings (R, Tableau).

Evaluators Also Use Big Data.  Large secondary datasets are appropriate in needs assessments and for measuring population-level outcomes. Community-level data, or data available for small levels of geography, provide context and can be used to derive neighborhood indicators. Evaluators must be able to not only access and manipulate this and other kinds of Big Data but to ultimately learn to use data science to maximize the value of the data.

Rad Resource: The American Community Survey (ACS)  is an especially rich, although recently controversial, Big Data resource for evaluators. The survey offers a wide range of data elements for areas as small as the census block and as specific as the percent of carpoolers working in service occupations in a census tract.

Hot Tips:

Rad Resource: The Census Bureau’s OnTheMap application is an interactive web-based tool that provides counts of jobs and workers and information about commuting patterns that I explored in an AEA Coffee Break webinar.

Lessons Learned: Data Science is Storytelling: Below is a map of unemployment rates by census tract from the ACS for Baltimore City and surrounding counties.  This unemployment data is overlaid with data extracted from OntheMap depicting job density and the top 25 work destinations for Baltimore City residents.  The map shows that 1) there are high concentrations of unemployed residents in inner-city Baltimore compared to other areas, 2) jobs in the region are concentrated in Downtown Baltimore and along public transportation lines and the beltway, and 3) many Baltimore City workers commute to areas in the surrounding counties for work.  Alone, these two datasets are robust but their power lies in visualizing data and interpreting relevant intersections between them.

Stewart map

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I’m Taj Carson, the President of Carson Research Consulting (CRC) in Baltimore, MD. CRC is a research and evaluation consulting firm and we’ve seen first-hand how neighborhood-level data is increasingly being used for tasks such as identifying community conditions and trends or measuring population-level outcomes in research and evaluation.

Recently, I was in a meeting where a group of human service providers met to discuss where to locate a program for pregnant and parenting teen mothers in Baltimore City. They realized that what they really needed to know was— which communities have the highest teen birth rates in Baltimore?  A colleague went into a storage room and retrieved a huge, styrofoam-backed map from 2008, showing teen birth rates across the city, so the group could decide where to locate the program. While I was impressed with the fact that they valued data enough to use it in the early planning stages, and that they were actually able to remember where they had stashed a map from 2008, I was encouraged to get the word out about the DataMind as an interactive mapping tool that would allow them to see this information without rummaging around in a closet.

Lessons Learned:

  • Creative, forward thinking program planners know when to bring data to the table to make decisions.
  • While there is a wealth of available data on our communities, we are short on ways to visualize that information on the spot and identify patterns across different parts of the city.
  • Visualizing several data sources together in an interactive map allows for a more complex understanding of data, even if it is on paper.

Rad Resource: The Baltimore DataMind is an interactive mapping tool that allows users to visualize data for Baltimore City neighborhoods to promote collaboration, advocacy, informed decisions, and effective policy making. Users can compare data across neighborhoods, create a community profile of a neighborhood, and combine data indicators and community resources and assets in one map. In the above-described situation, the Baltimore DataMind could have provided the information this group needed by looking at the “Teen Birth Rate” data in the Children and Family Health Indicators section of the map. They also could have mapped out the location of schools and community health centers in those neighborhoods. Hot Tips:

  • You Don’t Have to be a GIS Expert: Neighborhood-level data could have easily been pulled up in theBaltimore Datamind “widget” in an easy-to-use interface developed by Policy Map.
  • Data can be shared: The maps can be printed and then shared with neighborhood stakeholders, funders and community residents.

Baltimore DataMind 2

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

I’m Susan Kistler, AEA’s Executive Director, and contributor for each Saturday’s aea365 post. This week I’m writing on Thursday and queuing it up for Saturday, anticipating a couple of busy days ahead. I also serve as a houseparent at a small Boarding school on the south coast of Massachusetts, known as “the School by the Sea.” We’re right on the water and the school’s assets include a 92 foot schooner and a slew of other boats. It’s looking like we’re in the predicated path of hurricane Irene and that means the disaster preparedness plan goes into action and we’re looking at two days of hauling boats and moving the contents of labs and classrooms and homes to higher ground.

Lesson Learned: I’m fascinated by major storms and watching the radar and predictive models that strive to anticipate their path. I’ve written before about how thinking evaluatively is a way of life, and storm tracking and response brings that to the fore. Rather than succumb to the hype “the storm of the century is coming your way,” evaluative thinking based on data and experience from previous events allows us to take action, respond to changing conditions, and stay safe.

As I was keeping an eye on the storm (it is soon to pass over my mother’s house in the Bahamas), I was reminded of other mapping resources of use to evaluators.

Rad Resource – MapAction Field Guide to Humanitarian Mapping: This past month, MapAction released a free (thanks to a Dulverton Trust grant) updated version of this guide. While the context is guidance for humanitarian organizations, and thus would be particularly useful for those working in international M&E, its articulate explanations of the fundamentals of GIS, using google earth for mapping, and data sources would be valuable to anyone considering a mapping project, including those undertaking community mapping endeavors.

Rad Resource – Google Earth and Mapping Resources: This curation tree provides links to a range of google earth and mapping resources. To make the most of it:

  • Hot Tip: Note that the software is flash-based and thus doesn’t work on iPad
  • Hot Tip: Drag the ‘curated by’ box down and to the left to get it out of the way
  • Hot Tip: Click on any of the nodes (the outer circles) to view the referenced site and, once on the site, you can use the “Next” box in the upper right to scroll through all of the recommended sites without returning to the curation tree

Rad Resource – Intrdouction to GIS and Spacial Analysis in Evaluation Workshop: Arlene Hopkins and Stephen Maack will be offering a mapping workshop specifically for evaluators at AEA’s 2011 annual conference this November in Anaheim.

The above opinions are my own and do not necessarily represent those of the American Evaluation Association. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · ·

AEA365 began on January 1, 2010. Before we promoted this resource, we reached out to dedicated authors who believed in the project in order to populate the site with starter content. Those who contributed in week 1 wrote for an audience of fewer than 10. One year later we have over 1500 subscribers and are re-posting the contributions from those trailblazers in order to ensure that they receive the readership they deserve.

Hello my name is Tarek Azzam and I am an assistant research faculty member at Claremont Graduate University. I am interested in incorporating geography into our evaluation analysis, and reporting. I’ve found many tools that can help us do this through the use of GIS enabled websites like maps.google.com and many-eyes.com.

Hot Tip: you can use google to create maps that help you represent the location of different program sites, within these maps you can include information about the program performance through written descriptions or by color coding the icons to indicate high or low levels of performance (e.g. red icons=poor performance, green icons=excellent performance). Google maps also provides a street view of the neighborhoods that you can use to get a sense of the community condition. Website: maps.google.com

Hot Tip: many-eyes.com is a website that allows you to create interactive GIS maps with data you collect. For example, you can map changes in academic performance across various counties in a state by importing this data from an excel sheet into Many-eyes.com. The GIS software will create a color coded map that shows you how different counties performed on this indicator (typically the darker the color the higher the performance). This gives you a quick visual of how the program might have influenced different geographic regions (and may drive additional inquiry). Website: many-eyes.com

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Tarek Azzam on Geographic Information Systems (GIS)

· ·

My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI (http://www.esri.com/) often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at: http://gislounge.com/open-source-gis-applications/

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

· · · ·

Hi, my name is Bi Deng. I am a doctoral student at Claremont Graduate University (CGU) studying Organizational Behavior with a co-concentration in Evaluation. Back in 2009, I attended a CGU workshop by our GIS specialist Warren Roberts and was blown away by the utility of GIS in social science research, particularly for evaluation projects. I’ll be talking about how GIS can help us answer a wide range of place-, space-, and time-oriented questions (Meeks, 2008).

What is GIS? A geographic information system (GIS) captures, stores, analyzes, manages, shares, and presents data that is linked to location, such as people to an address or crime to a demographic area. GIS allows us to view, interpret, and visualize data in ways that reveal relationships, patterns, and trends in the form of maps, globes, reports, and charts (ESRI.com).

Why use GIS? If you were to look at a tabular set of data, it is not easy to interpret or see a pattern in a large database. GIS allows you to visualize large amounts of complex, spatial data across layers of various maps. Most problems and issues (e.g., cultural, environmental, economic, political, social) we face today exist in a geographic context (ESRI.com). An estimated 80% of all data held by business and government organizations have a spatial component (Franklin, 1992) – which means most data in can be analyzed “spatially.”

Some of the basic research questions GIS can help us answer include (W. Roberts, personal communication, January 23, 2009):

  • Location: What is at…?
  • Trends: What has changed…?
  • Condition: Where is the location that meets these criteria?
  • Pattern: What is the spatial pattern or distribution…?
  • Modeling: What if…?

For example, when doing spatial analysis, you can run queries to:

  • Identify specific features (e.g., what is the income level of a particular region that your client is serving)
  • Identify features based on conditions (e.g., out of a group of regional sites, you can identify all the high-performing or low-performing sites of your client)
  • Interactions (e.g., investigate whether poverty levels of a region affects the participation rates of your client organization)

Rad Resource: If you would like to learn more about GIS, the ESRI website is a great resource. ESRI’s ArcGIS community posts featured maps and applications of GIS. On ArcGIS.com, you can take a quick tutorial and register for a free account to create and share an interactive map: http://www.arcgis.com/home/gallery.html

Rad Resource: Want to explore what kinds of free data there is for researchers? Check out these sites:

Rad Resources:

  • Franklin, C. (1992). An introduction to geographic information systems: Linking maps to databases. Database, 15(2), 17-22.
  • Meeks, W. L. (2008). The utility of geospatial data and information used in Geographic Information Systems (GIS): An exploratory study into the factors that contribute to geospatial information utility. Unpublished doctoral dissertation, George Washington University, Washington, DC.

Cool Trick: Want to see a neat example of GIS in action? Check out Recovery.com – this site uses GIS to track and display data related to how funds of the Recovery Act is being awarded and spent. You can check to see how $275 billion of federal funds made available for federal contracts, grants, and loans are being used in your own community. http://www.recovery.gov/Pages/home.aspx

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Older posts >>

Archives

To top