AEA365 | A Tip-a-Day by and for Evaluators

TAG | Geographic information systems

Hello – I’m MaryAnn Sorensen Allacci, Director of Projects for Environmental Health, Knowledge, & Action, Inc. We provide a wide range of evaluation and advocacy services that focus on improving people’s well-being by improving their environments.  As a result, we will often work on community environmental health issues that require our consideration of many levels of organization from different perspectives.  One method we have used is geographic information systems (GIS) and spatial data analysis to understand what is going on in a particular area and how the phenomena are distributed across the landscape.

Hot Tip: When looking at and using spatial or geographic locational data, particularly in evaluation, it is important to be careful about mixing different levels of analysis and expecting that what you find on one level (e.g., a larger scale of data collected) can and will apply to another (i.e., smaller) level or scale of measurement. This is often referred to as “ecological fallacy.”

For example, if you want to look at census tract data, a common source of spatial data, for education level of residents or, say, numbers of people receiving food stamps, the information you obtain at the census tract level may not tell you about specific neighborhoods that lie within the census tract. Census tract data are aggregates of information from smaller units, i.e., census blocks and block groups, and could have different outcomes at the individual block or even street segment level.  Data reported at the census tract level, the larger level of analysis, could mask the phenomena at the smaller neighborhood level by leading the reader to mistakenly assume that the census tract figure is a valid representation of what is going on in the smaller units.  The number of preschoolers who could benefit from an early intervention educational program might be concentrated largely in one less-advantaged neighborhood while the overall census tract count is considerably lower than other tracts.

Cool Trick: As a rule of thumb, for purposes of validity you want to seek data as close to the level of analysis in which you are interested. Sometimes that data can be difficult to find, but you may be surprised at the kinds of data ready availability from different sources in the public realm.  Nested or multilevel studies deal with the complications of different levels of data being represented twice, or co-correlation, using specific study designs and statistical management.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI ( often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at:

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

· · · ·

My name is Sarah Hug and I am a Research Associate with the Alliance for Technology, Learning and Society (ATLAS) at the University of Colorado at Boulder. I am going to give a few tips regarding “pipeline evaluation” for programs aimed at increasing enrollment and diversity in science, technology, engineering, and math (STEM) fields.

Pipeline programs aim to change the career trajectories of young people in the long term. These goals for youth are often beyond the scope of the evaluation and program timelines- for example, an academic science program targeting middle school students will not be able to collect career data for at least five years, when students have graduated high school. What should  an evaluator study in the meantime?

  1. Student aspirations and interest: Evaluators can focus on student interest in the STEM fields, and their changing or continuing aspirations for STEM careers.
  2. Student knowledge of the fields: Knowledge about careers in technical areas is essential for advancement in STEM, particularly for underrepresented and under-resourced students. Evaluators can focus efforts on program participants’ change in career awareness at all academic stages. Some elements of career awareness evaluators might measure include:  knowledge of the depth and breadth of science careers, knowledge regarding what scientists do, and familiarity with the level of education needed to attain specific careers.
  3. Students’ “next- step behaviors”: Students’ college and career readiness can be influenced by early academic experiences. Discover what students might do at each level of the academic pipeline to further their STEM career readiness. This is often context and even school district specific-for example, are there clubs or camps students might join to advance their careers? What high school course choices could indicate students’ preparation for STEM careers? It is particularly important to consult program directors and school partners to gather ideas for measuring “next-step behaviors”. Comparing your program findings to local or national data for similar groups is essential for showing program impact.

Rad Resource: The National Center for Educational Statistics provides national data on student enrollment, graduation, and academic behavior at all levels of the education pipeline. Check out “The Condition of Education 2010” report, and data exploration tools to understand how students are progressing at different stages of the pipeline.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to

· ·

Greetings fellow AEA365 blog readers. My name is Duncan Meyers and I am a graduate student at the University of South Carolina in the Clinical/Community Psychology program. I have a strong interest in evaluation and have been an evaluator for community-based mental health services, an after-school obesity prevention program, and a project aimed at increasing walking through environmental strategies in underserved communities. Within the last three years the projects I have been involved with have utilized geographic information systems (GIS) to enhance our evaluations. A GIS is a system which consists of computer software and data; specifically, geospatial data (i.e., entities or events that can be described in a geographic fashion). These systems can be used to view and manage information about geographic places which are important to your evaluation (e.g., states, counties, communities, schools, etc.) and analyze spatial relationships.

Given the ways in which GIS has benefitted the projects that I have been involved with, I would like to share a helpful tip and point you toward a free resource that I use very frequently.

Hot Tip: First and foremost – like any software assisted analytic tool – GIS involves a learning curve and it is unlikely that evaluators will be able to sit down and use it right away without any training. However, utilizing professional contacts you may have is a great way to work around the learning curve. As long as a clear research question is identified, these contacts may be able to help you visually display your data and analyze relationships among your data and test hypotheses. If you don’t already have such contacts, many county, state, and national agencies/organizations have GIS offices that may be willing to collaborate with you. Also, if there is a University near you the Geography department can be a great help.

Rad Source: One of the most helpful – and basic – functions I have used with GIS is geocoding. Geocoding finds the latitude and longitude coordinates of an address. These coordinates are essentially x and y coordinates that you can put on a map. Here’s a free geocoder that I use a lot: allows you to use an Excel spreadsheet to convert addresses into map coordinates, and then create maps that you can view. Download the template, add some addresses, and hit the “map now” button. Here’s a YouTube tutorial to help get you started:

Geocoding is a basic yet integral step in setting up a GIS analysis. Linking the maps you create to your evaluation will take additional steps, and if you are interested in getting some ideas be sure to tune into the AEA Coffee Break Webinar on June 3rd!

· · ·

My name is Juan Paulo Ramírez and I am a research specialist with the University of Nebraska Public Policy Center. I use Geographic Information Systems (GIS) for project evaluations which have included a broad variety of applications both in the social and physical sciences. Recently a lot of interest has been concentrated on geographic visualization, in particular the integration of geobrowsers like Google Earth with commercial GIS software such as ArcGIS and others. This allows distributing GIS data to many who would not necessarily use GIS software but are consumers of geospatial data. The good news is that the use of GIS software and the data associated to it has become easily accessible to the evaluation community at a very reduced cost and sometimes with no costs at all!

Hot Tip: Check out the YouTube videos posted by ESRI, the California based enterprise that created ArcGIS. Search for “ESRI TV.” If you are a neophyte to GIS and in particular to the ArcGIS family, these tutorial videos will save you thousands of dollars in training. Even if you have experience in using ArcGIS, these tutorials will demonstrate new tricks that will enhance your analytical capabilities.

Cool Trick: The newest versions of GIS software, including ArcGIS, have incorporated exporting features for Google Earth format files, known as “KML” format. This will allow you to distribute your GIS information (points, lines or polygons) along with databases to your stakeholders, colleagues, and community in general who do not have GIS software. The only requirement is that your recipients must install the free version of Google Earth in order to read KML format files, and the geobrowser will automatically display all the geospatial information and associated databases that you sent to them (i.e., attached in a e-mail). To see the new features of Google Earth, check:

Rad Resource: Want to learn more about GIS and spatial visualization? The following book explores the theory behind geographical visualizations, including examples of map animation, and geovisualization tools, and provides insights to the future development of geographic visualization: Dodge, M., McDerby, M., & Turner, M. (Ed.). (2008). Geographic visualization: Concepts, tools and applications. West Sussex, England: John Wiley & Sons, Ltd.

Want to learn more from Juan Paulo? He is offering a Coffee Break Webinar on Using Google Earth for Evaluation: Applications in Environmental Evaluation and Beyond this Thursday. This is free for AEA members and a paid pass is available for nonmembers. Learn more at

· · · · ·

My name is Susan Kistler and I am the Executive Director for the American Evaluation Association. I contribute each Saturday’s post to the aea365 blog. This week I am writing from Atlanta at the Nonprofit Technology conference.

Do you work for or with nonprofit organizations? Have you experienced challenges due to financial constraints that make technology purchases for evaluation beyond the budget?

Hot Tip: Take a look at TechSoup, the “technology place for nonprofits.” TechSoup has resources, training, a peer-to-peer community, and a donated technology program – TechSoup Stock. Their donated tech program gives nonprofits access to products from a range of big name (and not so big name) companies. Examples include the full Microsoft Office Suite including Access and Excel; ArcGIS from ESRI for spatial analysis; and Crystal Reports from SAP for data visualization and reporting. And the cost? Each product has an administrative fee, but most are well below even discounted retail prices. As an example, the full Microsoft Office 2007 suite is $20. Organizations do need to go through a relatively painless qualification process, and the eligibility criteria vary from product to product, but the resource is definitely worth checking out.

The opinions expressed above are my own and do not necessarily represent those of my employer, the American Evaluation Association.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to

· · · · · · ·

Hello my name is Tarek Azzam and I am an assistant research faculty member at Claremont Graduate University. I am interested in incorporating geography into our evaluation analysis, and reporting. I’ve found many tools that can help us do this through the use of GIS enabled websites like and

Hot Tip: you can use google to create maps that help you represent the location of different program sites, within these maps you can include information about the program performance through written descriptions or by color coding the icons to indicate high or low levels of performance (e.g. red icons=poor performance, green icons=excellent performance). Google maps also provides a street view of the neighborhoods that you can use to get a sense of the community condition. Website:

Hot Tip: is a website that allows you to create interactive GIS maps with data you collect. For example, you can map changes in academic performance across various counties in a state by importing this data from an excel sheet into The GIS software will create a color coded map that shows you how different counties performed on this indicator (typically the darker the color the higher the performance). This gives you a quick visual of how the program might have influenced different geographic regions (and may drive additional inquiry). Website:

This contribution is from the aea365 Daily Tips blog, by and for evaluations, from the American Evaluation Association. Please consider contributing – send a note of interest to



To top