AEA365 | A Tip-a-Day by and for Evaluators

TAG | census

Hello, Erin M. Liang, Mark M. Holske, and Humberto Reynoso-Vallejo here, members of the research team evaluating the Health Care Cost Containment law (Chapter 224) from the Office of the State Auditor.

Using administrative data with a significant number of missing key variables (e.g. race/ethnicity) can be challenging in trying to answer specific evaluation questions. One of our research questions seeks to evaluate the impact of Chapter 224 on racial/ethnic disparities in health outcomes. Both administrative data sets available to us (the Massachusetts Medicaid Program (MassHealth) and the All Payer Claims Database (APCD)) have sparsely populated information about race and/or ethnicity. Since we were not able to apply imputation techniques due to the large number of missing values, we have to use alternative methods. As a proxy for race/ethnicity data, we used US Census Bureau Data and GIS mapping software.

Hot Tip: Use state level census data to account for missing racial/ethnic values. The US Census collects detailed race/ethnicity data at the state level every 10 years. Census data is publicly available on the American Fact Finder website. There are several data sets available, and this tool can aid in determining which data set best suits the needs of your project.

In this project, census data from 2010 will be applied to collected data spanning years 2006 to present, segmented by ZCTA (zip code tabulation area). Researchers should be aware that a zip code can be in more than one ZCTA. To learn about how zip codes are translated to ZCTAs, consult this census demonstration.

Large scale

This map was made using QGIS and census data. The map is segmented by cities/towns with the darker areas representing populations with a higher percentage of diverse residents.

Rad Resource:

  • The Census Bureau Help Line is 1-800-923-8282. A census representative can help to download the correct data.
  • In addition to QGIS, a large project with sufficient funding could utilize Tableau, an easy to use data analysis tool that allows users to visualize data with graphs, cross-tabs, and maps. While these functions can be done in other tools like Excel or ArcMap, Tableau allows a user to create a dashboard containing all visualizations on the same page.

The American Evaluation Association is celebrating Large Scale Evaluation Week. The contributions all this week to aea365 come from evaluators who have worked on the evaluation of the Health Care Cost Containment Law in Massachusetts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Jennifer Sullivan Sulewski and I am a Research Associate at the Institute for Community Inclusion (ICI), University of Massachusetts Boston and past co-chair of AEA’s Disabilities and Other Vulnerable Populations (DOVP) TIG.

At ICI, we do a lot of work involving analysis of publicly available datasets to determine outcomes and trends for people with disabilities at the state and national levels. These data can be used to assess areas of need, establish baselines, and track progress over time for outcomes such as employment rates, economic status, and educational attainment. My post today highlights a couple of particularly useful data sources.

Hot Tip:

  • Pay attention to how disability is defined in your data source.  Each system or survey is likely to have a different set of disability categories and definitions. For example, the Census Bureau determines if respondents to the American Community Survey have a disability by asking if they have any of six specific conditions or functional impairments (http://www.census.gov/people/disability/methodology/acs.html). The Social Security Administration defines disability as a long-term impairment affecting the ability to work. Other systems (such as Vocational Rehabilitation and developmental disabilities services) have their own ways of assessing disability status and eligibility for services.

Rad Resource:

  • U.S. Census Bureau. The Census Bureau website includes a wealth of searchable data and customization reports on both the decennial census (last conducted in 2010) and more frequent data collection efforts such as the American Community Survey.
  • Statedata.info. This website compiles data on employment outcomes and other population statistics for people with disabilities nationally and state by state, using data from state intellectual/developmental disabilities agencies, the Rehabilitation Services Administration, the U.S. Department of Labor, the Social Security Administration, and the Census Bureau. In the interest of full disclosure: statedata.info is developed and maintained by my team at the ICI.

 

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

Hi, I’m Sara Plachta Elliott, Evaluation Fellow at the Skillman Foundation in Detroit, Michigan, through a grant to Brandeis University’s Center for Youth and Communities. As an Evaluation Fellow, I work with Foundation staff to create learning opportunities that inform the work of improving school and neighborhood conditions for Detroit’s kids.In 2010, Brandeis conducted a System of Supports and Opportunities (SOSO) analysis in six neighborhoods where Skillman focuses its grantmaking and changemaking efforts. Data collection involved interviewing youth program and basic service provider staff to assess program offerings, participation rates, program quality, and accessibility.

Below are learning opportunities that were created with SOSO data.

Lesson Learned: Have stakeholders review findings before the final report is prepared.

  • An internal review with Foundation staff helped identify inaccuracies in participation rates and site location addresses.
  • Then two-page SOSO snapshots for each neighborhood were released.
  • Key neighborhood and youth program stakeholders were engaged in a review of findings and lifted up questions about how data were collected. This review process ensured “on the ground” validity of the analysis.
  • For full transparency, the dataset was also released to partners for system planning purposes.

Hot Tip: Create maps or other visual products.

The Foundation engaged its partner Data Driven Detroit to create maps of agency and program site locations. In meetings with stakeholders, we reviewed these maps along with youth population maps. In one neighborhood, we learned that most kids lived on the west side but program sites were clustered on the east side. The collective “ah ha” moments helped the Foundation and its stakeholders work together to fill gaps.

Consider hosting a discussion session and ask community partners and stakeholders to interpret brief data reports and maps. What patterns do they see in the data?

Cool Trick: Engage youth and residents in reviewing data, not just program staff.

In the summer of 2011, a social service agency, Southwest Solutions, organized a community youth mapping project. Youth reviewed the Brandeis SOSO data, then designed and conducted their own neighborhood opportunities survey.

Youth researchers were paid through summer youth employment funding and walked every street in their neighborhood twice, administering surveys to youth, youth program staff, and business owners, as well as mapping vacant properties. They discovered that some local businesses wanted youth to work during school hours, thereby tempting them to drop out of school. The youth also learned that students wanted more college and career preparation opportunities, mirroring findings from the SOSO report that more of these opportunities were needed.

Evaluation reports can sit on a shelf if not accompanied by intentional learning opportunities. As an evaluator, encourage clients to create interactive learning opportunities with short, visual reporting products. Spark their collective learning!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · · · · · ·

My name is Melissa Biel and I am an independent consultant. I frequently work with hospitals, clinics and community organizations to conduct community health needs assessments. A common component of a community needs assessment is identifying access to primary health care. I have found the Uniform Data System (UDS) and the associated UDS Mapper to be useful resources to assess access to health care.

Rad Resource #1: Did you know that all Federally Qualified Health Centers (FQHCs)/Community Health Centers are required to collect and report data to the Bureau of Primary Health Care? The Uniform Data System (UDS) tracks a variety of information, including patient demographics, services provided, staffing, clinical indicators, utilization rates, costs, and revenues. UDS data are collected from Community Health Centers and reported at clinic, state, and national levels. http://bphc.hrsa.gov/healthcenterdatastatistics/index.html

Hot Tip: On the Bureau of Primary Health Care’s Health Center Data webpage, check out the navigation options on the left side of the page for some useful data tools. Demographic trends can be shown by category and state, and data can be compared by year and by state or national geography.

Rad Resource #2: The UDS Mapper presents data from the UDS reports in map and table format. To access the UDS Mapper, you must register for a free account. Data can be viewed by Zip Code, ZCTA, County or State. The UDS Mapper incorporates 2010 Census data with Community Health Center report data. By mapping the total population, low-income population, and number of residents served and unserved by Health Centers, I am able to identify the percentage of the population that lacks access to primary health care in a given geographic area.

Hot Tip: In addition to colorful, detailed maps, the site offers a table tab that shows the selected geographic area data in table form. The site has functions that allow the addition of lines, shapes and text to the maps. The maps can be saved as a pdf and the tables saved as an Excel spreadsheet or pdf. Both maps and tables can be emailed through a site-generated URL.

Hot Tip: Move your cursor over the map and detailed pop up boxes with Census and Health Center data appear on the map at the Zip Code level. The map also shows overlays of Health Professional Shortage Areas (HPSA) and Medically Underserved Areas (MUA).

Hot Tip: While on the UDS Mapper website, click on the Other Resources tab for access to tools and data, articles and references, and to download state and county data.

UDS Example Map

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators

· ·

Hello – I’m MaryAnn Sorensen Allacci, Director of Projects for Environmental Health, Knowledge, & Action, Inc. We provide a wide range of evaluation and advocacy services that focus on improving people’s well-being by improving their environments.  As a result, we will often work on community environmental health issues that require our consideration of many levels of organization from different perspectives.  One method we have used is geographic information systems (GIS) and spatial data analysis to understand what is going on in a particular area and how the phenomena are distributed across the landscape.

Hot Tip: When looking at and using spatial or geographic locational data, particularly in evaluation, it is important to be careful about mixing different levels of analysis and expecting that what you find on one level (e.g., a larger scale of data collected) can and will apply to another (i.e., smaller) level or scale of measurement. This is often referred to as “ecological fallacy.”

For example, if you want to look at census tract data, a common source of spatial data, for education level of residents or, say, numbers of people receiving food stamps, the information you obtain at the census tract level may not tell you about specific neighborhoods that lie within the census tract. Census tract data are aggregates of information from smaller units, i.e., census blocks and block groups, and could have different outcomes at the individual block or even street segment level.  Data reported at the census tract level, the larger level of analysis, could mask the phenomena at the smaller neighborhood level by leading the reader to mistakenly assume that the census tract figure is a valid representation of what is going on in the smaller units.  The number of preschoolers who could benefit from an early intervention educational program might be concentrated largely in one less-advantaged neighborhood while the overall census tract count is considerably lower than other tracts.

Cool Trick: As a rule of thumb, for purposes of validity you want to seek data as close to the level of analysis in which you are interested. Sometimes that data can be difficult to find, but you may be surprised at the kinds of data ready availability from different sources in the public realm.  Nested or multilevel studies deal with the complications of different levels of data being represented twice, or co-correlation, using specific study designs and statistical management.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Virginia Dick and I am currently public service evaluation faculty at the Carl Vinson Institute of Government at the University of Georgia. Most of my work focuses on assisting state and local government agencies, and other university divisions, with evaluation of programs, policies and systems.

As part of my role I often find myself working with a wide range of individuals with different backgrounds, perspectives, purposes, and information assessment styles. It has been important to find ways to help different groups examine and understand relevant evaluation data using a wide range of mechanisms.

Most recently, I have begun working with our state child welfare agency to use GIS (Geographic Information Systems) methods to examine child welfare client characteristics and outcomes spatially through mapping. Often key stakeholders (community members, agency leadership, and social work students) have expressed new and interesting perspectives and interpretations of the data when it is portrayed via mapping rather than in traditional charts and tables.

Rad Resource: ESRI (http://www.esri.com/) often provides free training and educational opportunities to work with their mapping software and may be available through some universities.

There are many open source software options out there, some of which I am currently working with at the University of Georgia Information Technology Outreach Service to explore with my current project. A list of open source options is available at: http://gislounge.com/open-source-gis-applications/

Hot Tip: When working with a group reviewing the data and relationships between variables, start with a few layers and options on the map and slowly build and add additional components as the individuals start to become more comfortable talking about the relationships between the different variables.

Hot Tip: By looking at census tracts as units it allows groups to discuss the relationship between variables without having to dig down to the individual street address level which can become much more complicated when compiling the maps. Often analysis at the census tract level can be most beneficial to communities and government agencies rather than the individual street address level.

Hot Tip: Let the stakeholders generate the ideas and discussion among themselves to get the richest information about the perceived relationship between variables. This is particularly useful when looking at small units such as counties or smaller (with the mapping done at the census tract or block level).

Want to learn more about Virginia’s work using GIS? Come to the poster exhibition on Wednesday evening in San Antonio this November for AEA’s Annual Conference.

· · · ·

Archives

To top