AEA365 | A Tip-a-Day by and for Evaluators

TAG | mapping

I am Scott Chazdon, Evaluation and Research Specialist with the Extension Center for Community Vitality, University of Minnesota. I have gained skills in a process known as Ripple Effect Mapping (REM) to document impacts of Extension community development programs. REM sessions often spur important thinking, connections and work.

REM is a participatory group method that engages program and community stakeholders to retrospectively and visually map the chain of effects resulting from a program or complex collaboration. The REM process combines elements of Appreciative Inquiry, mind mapping, group interviewing, and qualitative data analysis. It is a powerful tool for documenting both the intended and unintended results of a program. It is also a way to engage and re-energize program participants and stakeholders around shared goals.

Rad Resource: A more in-depth introduction to REM is at University of Minnesota Extension feature article on REM – “Ripple effect mapping makes waves in the world of evaluation”

Lesson Learned: What started as a great method for evaluating community leadership programs morphed into a tool for a broad range of programs.

In Minnesota, an effort to document the impact of urban Master Gardeners working in the neighborhoods became a more inclusive and community-driven project that showcased the many different outcomes of the program that may have been overlooked.   Here is a thumbnail graphic of the core section of the Ripple Effect Map from that project.

Rad Resource: You can find full-sized REM graphics at this site University of Minnesota Extension REM Blog

AEA2

Lesson Learned: Recruiting the right number and mix of people is crucial in Ripple Effect Mapping. In terms of numbers, these are larger than focus groups, but if you go beyond 20 people you may not be able to include all voices in the process. I prefer groups of 12 to 20 people.

You can invite both direct participants and non-participant stakeholders. This non-participant group can include funders, local elected officials, other influential figures, or representatives of the media.

Lesson Learned: This mix of people creates an insider-outsider dynamic that sometimes leads to game-changing insights about efforts that have already happened, as well as efforts that could happen! That’s why Ripple Effect Mapping makes sense as a developmental evaluation tool.

Rad Resources: To find out more about REM and approaches that can be taken, as well as if might be a tool you can use, take a look at these two articles: 1) Journal of Extension — Using Ripple Effect Mapping to Evaluate Program Impact: Choosing or Combining the Methods That Work Best for You and 2) Journal of Extension — Ripple Effect Mapping: A “Radiant” Way to Capture Program Impacts

The American Evaluation Association is celebrating Extension Education Evaluation (EEE) TIG Week with our colleagues in the EEE AEA Topical Interest Group. The contributions all this week to aea365 come from our EEE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, we are Miranda Yates and Anne Gleason from the Program Evaluation and Planning (PEP) Department at Good Shepherd Services in New York City.  As part of our agency-wide commitment to youth and family development and evidence-based practice, we recognize the profound value of partnering with youth participants at all stages of program evaluation.  Our approach supports youth to identify powerful questions, select evaluation methods, collect and analyze data, interpret findings, and carry out plans to strengthen programs.  In particular, we have found that focus groups with youth participants are an important means of clarifying the processes though which programs make a difference.

Hot Tips:

Inspired by all we learned at the 2012 summer session of the Critical Participatory Action Research (CPAR) Institute, we use two particular techniques to engage participants in our program evaluation process.

1.  MAP ACTIVITY.  We start focus groups by passing out blank paper and colored pens and asking everyone to spend 10 minutes visually representing their experience in a particular Good Shepherd program.  We then have all youth in the group share what they depicted. This has proven to be a great way to get everyone engaged in conversation, surface common themes and metaphors, and better understand the program’s impact.  Below are two examples of maps drawn by participants in our educational programs which assist over-age and under-credited youth in earning their high school diplomas and successfully transitioning to college.  Both maps highlight the difference made by strength-based programming and an emphasis on social support.

Capture

 2.  INTERPRETING SURVEY FINDINGS.  Another engaging activity is to present program data to participants and ask for their help interpreting it.  PEP administers a number of surveys to participants to assess changes in participant behaviors, attitudes or knowledge. Understanding that we can be limited in our ability to interpret data findings from youth surveys, we recently asked a group of youth to help us interpret results from the Youth Experience Survey.  The youth offered critical insights into why certain aspects of the program were rated higher than others.  These were conclusions that we would not have come to on our own and has allowed for a much richer analysis of the survey data.

Rad Resource:

For more ideas on approaches to engaging groups in meaningful conversations, check out resources from People Potential.

The American Evaluation Association is celebrating Youth Focused Evaluation (YFE) TIG Week with our colleagues in the YFE Topical Interest Group. The contributions all this week to aea365 come from our YFE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Greetings! I’m Nichole Stewart, a doctoral student in UMBC’s Public Policy program in the evaluation and analytical methods track. I currently work as an analyst, data manager, and evaluator across a few different sites including Baltimore Integration Partnership, Baltimore Workforce Funders Collaborative, and Carson Research Consulting Inc.

Lessons Learned: The Growing Role of Data Science for the “Little” Data in Program Evaluation. Evaluators are increasingly engaged in data science along every step of the evaluation cycle. Collecting participant-level data and developing indicators to measure program outputs and outcomes is now only a small part of the puzzle. Evaluators are working with more complex data sources (administrative data), navigating and querying data management systems (ETO), exploring advanced analytic methods (propensity score matching), and using technology to visualize evaluation findings (R, Tableau).

Evaluators Also Use Big Data.  Large secondary datasets are appropriate in needs assessments and for measuring population-level outcomes. Community-level data, or data available for small levels of geography, provide context and can be used to derive neighborhood indicators. Evaluators must be able to not only access and manipulate this and other kinds of Big Data but to ultimately learn to use data science to maximize the value of the data.

Rad Resource: The American Community Survey (ACS)  is an especially rich, although recently controversial, Big Data resource for evaluators. The survey offers a wide range of data elements for areas as small as the census block and as specific as the percent of carpoolers working in service occupations in a census tract.

Hot Tips:

Rad Resource: The Census Bureau’s OnTheMap application is an interactive web-based tool that provides counts of jobs and workers and information about commuting patterns that I explored in an AEA Coffee Break webinar.

Lessons Learned: Data Science is Storytelling: Below is a map of unemployment rates by census tract from the ACS for Baltimore City and surrounding counties.  This unemployment data is overlaid with data extracted from OntheMap depicting job density and the top 25 work destinations for Baltimore City residents.  The map shows that 1) there are high concentrations of unemployed residents in inner-city Baltimore compared to other areas, 2) jobs in the region are concentrated in Downtown Baltimore and along public transportation lines and the beltway, and 3) many Baltimore City workers commute to areas in the surrounding counties for work.  Alone, these two datasets are robust but their power lies in visualizing data and interpreting relevant intersections between them.

Stewart map

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I’m Marybeth Neal, I am a cultural anthropologist that uses ethnographic methods to engage stakeholders in creating their own ongoing systems of evaluation.

This summer I had the absolutely delightful experience working with a brilliant group of teens and young adults who train both youth and adult members of social change organizations on how to create effective youth-adult partnerships.

They wanted to create a pre-test and then two post-tests (given immediately following the training and then six months later) to accompany their Youth Adult Partnership training.  The purpose of these tests were multiple:

  • To offer constructive feedback to the trainees and their organizations.
  • To document and share the results of the training with external audiences (funders and potential clients), and
  • For internal use (to improve the training and to customize it for particular audiences).

Together we created a simple logic model for the project, which came to be known as “the Grid.” The Grid functioned as a “sticky wall” where stakeholders could contribute their own ideas (using post-its).

Hot Tip:

We put the Grid on a wall of the office by the kitchen area – a popular spot!  It listed, in separate columns:

  • What are the desired learning outcomes of our training?
  • What are the training activities that teach these outcomes?
  • How do we know and how can we measure the extent to which participants and their organizations are learning what is being taught?

In addition, in the upper right hand corner, we asked:

Why do we want to create assessment tools?

In its simplicity, this visual representation of the project helped everyone to understand the complexity of the project and to contribute to it.  It was an efficient way to define key concepts used in the training, to make sure that each learning objective had a corresponding activity that taught the objective, and that we did not forget why we were doing this work.   Most importantly, it offered a way for the trainers’ to share their wisdom about how to recognize successful learning both for the individual trainees and for their organizations.

Rad Resource:  We used the Descriptive Question Matrix in The Ethnographic Interview by James P. Spradley to help us generate survey questions that could be used to measure changes in knowledge, skills and dispositions.  Spradley pioneered the teaching of ethnographic methods to undergraduates and this book and its companion book. Participant Observation, are simple, accessible and profound.

·

Hi, I am Robin T. Kelley and am an internal evaluator at a national nonprofit health organization that is funded by the Centers for Disease Control and Prevention to provide free capacity building assistance to HIV prevention organizations, health departments and their HIV planning groups.

In the HIV/AIDS field, there are a number of changes occurring; here are just a few major ones:  In 2010, there was the release of the U.S. National HIV/AIDS Strategy. All funded entities are now striving to align themselves with the major goals of this strategy. As of  2011, scientific studies that showed the effectiveness of adherence to HIV medicine in reducing the viral loads,  resources are placed into, biomedical interventions and the  emphasis is now placed  more on organizations conducting  high impact HIV prevention.

Lessons Learned:

One key method of building an organization’s ability to manage complex situations, particularly small organizations that serve vulnerable populations, or populations of color-is to strengthen their change management leadership skills.  Research has shown that in times of complexity, such as shifting federal and health priorities, organizations, businesses that serve minorities  often shut their doors first ,leaving underserved communities abandoned and without services.  To sustain these agencies, evaluators as well as program managers should be agile and flexible in understanding the community needs, their resources, staff strengths as well as weaknesses-to best manage the changes.

Hot Tips:

Here are some steps to take and useful tools to address HIV changes and changes in general:

1)     First, help the organization conduct an organizational diagnosis.  They must know what they have in order to consider what to change.

2)     Second, help the organization to conduct an environmental scan or asset mapping of their community to determine if there is still a need for their services.

3)     Then to help organizations to analyze the data.  Based on the findings, help the organization to do a SWOT analysis (an analysis of their strengths, weaknesses, opportunities and threats).  Depending on these findings,  perhaps  there is a way to merge efforts with another organization;

4)     Next, help the organizations communicate changes to all staff; without constant communication, rumors can fly and morale can sink.

5)     Finally, help the organization to create a process log so that they can record the number of new service requests and activities and to continue to justify their existence.

Rad Resources:

The American Evaluation Association is celebrating the Business, Leadership, and Performance TIG (BLP) Week. The contributions all week come from BLP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

My name is Gary Resnick and I am the Director of Research at Harder+Company Community Research, a California-based consulting firm. My background combines program evaluation with child development research, and I have an interest in system theory and networks.

Harder+Company has been involved evaluating First 5 programs in a number of California counties. First 5 arose from 1998 Proposition 10, adding a tax on tobacco products with funds distributed to counties to fund local programs that improve services for children birth to 5 and their families. An important goal of First 5 funding is to act as a catalyst for change in each county’s systems of care. To measure system change, we focused on inter-agency coordination and collaboration. Increases in coordination and collaboration would indicate that agencies are better able to share resources and clients, reduce redundancies and service gaps, and increase efficiency.

Rad Resource: The Levels of Collaboration Scale assesses collaboration, has excellent psychometric properties and can be administered in web-based surveys to agency respondents. To see it in action, check out this article in the American Journal of Evaluation. Originally a 5-point Likert scale, we combined the two highest scale points creating a 4-point scale to make it easier for respondents.

Hot Tip: Start by defining the network member agencies using objective, clear, and unbiased criteria. Later, you can expand the network by asking respondents to nominate up to three additional agencies with whom they interact.

Hot Tip: Select at least two respondents from each organization, three is better, from different levels of the organization, administrators and managers as well as direct line staff.

Lesson Learned: It is important to have complete, reciprocal ratings for each agency (even if not from all respondents). If you have too much missing data at the agency level, consider excluding the agency from the network.

Hot Tip: Use Netdraw, a Windows freeware program, to produce two-dimensional network maps from agency-level Collaboration Scale ratings. See our maps here. The maps identify agencies most involved with other agencies at the center of the map (key players) and those least involved, at the periphery of the network. Add attributes of agencies (e.g. geographic region served) to map subgroups of your network.

Hot Tip: Produce two sets of maps, one with no agency labels for public reporting, and another with agency labels, for internal discussions with clients and agencies. Convene a meeting with the agency respondents and show them the maps with agency labels, to help them understand where they stand in the network and to foster collaboration.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

I’m Johanna Morariu, a Director at Innovation Network, an evaluation consulting firm that works with nonprofit organizations and foundations.

Rad Resources: I want to share two extremely useful network analysis and mapping tools: Gephi and NodeXL. I use NodeXL for collecting, organizing, and analyzing network data and Gephi for attractively presenting sociograms or network maps.

In a past post, Shelly Engelman and Tom McKlin introduced NodeXL when they wrote about assessing the evolution of Social Networks Using NodeXL. In addition to the functionality they discussed, NodeXL can also be used to collect social network data from Twitter, Flickr, YouTube, and email, and NodeXL can open/read other network analysis file formats. (And with the recently released Social Network Importer, you can also work with Facebook social networks.) But for all its strengths and utility, the network maps that can be made in NodeXL leave something to be desired.

Lessons Learned: Visually observing network structural features is a critical component of network analysis. And for evaluation stakeholders to effectively discern features, it is important to create well-designed network maps—and that is exactly what Gephi does!

Gephi allows for unprecedented control and options while creating network maps. For example, groups of network nodes can be coded by color, or degree can be represented by increasing node size. Gephi also has the capacity to incorporate longitudinal data, to show changes over time.

Since a picture is worth a thousand words, here is the progression from NodeXL to Gephi drawn from a twitter search of the hashtag #eval on January 17, 2011.

First, the basic NodeXL map:

 the basic NodeXL map

After exporting the data from NodeXL to a GraphML file, uploading to Gephi, and tweaking, here is the new map:

And with another simple change (turning on automatic resizing by node degree), voilà!

Rad Resources: And since this is the DVRTIG week, I can’t help but share three other essential tools for creating visually appealing presentations:

  • Design Seeds for color palette inspiration.
  • Instant Eyedropper to get RGB values (for example, from a Design Seeds color palette) to use in visualizations.
  • Color Oracle to simulate color blindness to ensure visualization and design retain their meaning for every viewer.

Rad Resources: Interested in learning more about network analysis? Check out these great posts:

We’re celebrating Data Visualization and Reporting Week with our colleagues in the DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

·

My name is Melissa Biel and I am an independent consultant. I frequently work with hospitals, clinics and community organizations to conduct community health needs assessments. A common component of a community needs assessment is identifying access to primary health care. I have found the Uniform Data System (UDS) and the associated UDS Mapper to be useful resources to assess access to health care.

Rad Resource #1: Did you know that all Federally Qualified Health Centers (FQHCs)/Community Health Centers are required to collect and report data to the Bureau of Primary Health Care? The Uniform Data System (UDS) tracks a variety of information, including patient demographics, services provided, staffing, clinical indicators, utilization rates, costs, and revenues. UDS data are collected from Community Health Centers and reported at clinic, state, and national levels. http://bphc.hrsa.gov/healthcenterdatastatistics/index.html

Hot Tip: On the Bureau of Primary Health Care’s Health Center Data webpage, check out the navigation options on the left side of the page for some useful data tools. Demographic trends can be shown by category and state, and data can be compared by year and by state or national geography.

Rad Resource #2: The UDS Mapper presents data from the UDS reports in map and table format. To access the UDS Mapper, you must register for a free account. Data can be viewed by Zip Code, ZCTA, County or State. The UDS Mapper incorporates 2010 Census data with Community Health Center report data. By mapping the total population, low-income population, and number of residents served and unserved by Health Centers, I am able to identify the percentage of the population that lacks access to primary health care in a given geographic area.

Hot Tip: In addition to colorful, detailed maps, the site offers a table tab that shows the selected geographic area data in table form. The site has functions that allow the addition of lines, shapes and text to the maps. The maps can be saved as a pdf and the tables saved as an Excel spreadsheet or pdf. Both maps and tables can be emailed through a site-generated URL.

Hot Tip: Move your cursor over the map and detailed pop up boxes with Census and Health Center data appear on the map at the Zip Code level. The map also shows overlays of Health Professional Shortage Areas (HPSA) and Medically Underserved Areas (MUA).

Hot Tip: While on the UDS Mapper website, click on the Other Resources tab for access to tools and data, articles and references, and to download state and county data.

UDS Example Map

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators

· ·

I’m Susan Kistler, AEA’s Executive Director, and contributor for each Saturday’s aea365 post. This week I’m writing on Thursday and queuing it up for Saturday, anticipating a couple of busy days ahead. I also serve as a houseparent at a small Boarding school on the south coast of Massachusetts, known as “the School by the Sea.” We’re right on the water and the school’s assets include a 92 foot schooner and a slew of other boats. It’s looking like we’re in the predicated path of hurricane Irene and that means the disaster preparedness plan goes into action and we’re looking at two days of hauling boats and moving the contents of labs and classrooms and homes to higher ground.

Lesson Learned: I’m fascinated by major storms and watching the radar and predictive models that strive to anticipate their path. I’ve written before about how thinking evaluatively is a way of life, and storm tracking and response brings that to the fore. Rather than succumb to the hype “the storm of the century is coming your way,” evaluative thinking based on data and experience from previous events allows us to take action, respond to changing conditions, and stay safe.

As I was keeping an eye on the storm (it is soon to pass over my mother’s house in the Bahamas), I was reminded of other mapping resources of use to evaluators.

Rad Resource – MapAction Field Guide to Humanitarian Mapping: This past month, MapAction released a free (thanks to a Dulverton Trust grant) updated version of this guide. While the context is guidance for humanitarian organizations, and thus would be particularly useful for those working in international M&E, its articulate explanations of the fundamentals of GIS, using google earth for mapping, and data sources would be valuable to anyone considering a mapping project, including those undertaking community mapping endeavors.

Rad Resource – Google Earth and Mapping Resources: This curation tree provides links to a range of google earth and mapping resources. To make the most of it:

  • Hot Tip: Note that the software is flash-based and thus doesn’t work on iPad
  • Hot Tip: Drag the ‘curated by’ box down and to the left to get it out of the way
  • Hot Tip: Click on any of the nodes (the outer circles) to view the referenced site and, once on the site, you can use the “Next” box in the upper right to scroll through all of the recommended sites without returning to the curation tree

Rad Resource – Intrdouction to GIS and Spacial Analysis in Evaluation Workshop: Arlene Hopkins and Stephen Maack will be offering a mapping workshop specifically for evaluators at AEA’s 2011 annual conference this November in Anaheim.

The above opinions are my own and do not necessarily represent those of the American Evaluation Association. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · ·

Hello, my name is Juan Paulo Ramírez, independent consultant, sole owner of “GIS and Human Dimensions, L.L.C.” As many of you may know Google Analytics (GA) allows you to track down the number of visitors that a website receives during a certain period of time. But GA does a lot more than that. If you have installed the GA code into a website, GA offers a number of visualization tools that will allow you to analyze what is working and what is not working in your website, and ways to improve it. The following are two of the visualization tools that I like the most offered by Google Analytics:

Rad Resource: Google Analytics – Map overlay

Map overlay allows you to identify from where you are getting visitors. This is a great tool since it identifies your audience by geographic location and then potentially you can customize your website to the characteristics of that audience based on their demographics, culture, or interests. A coropleth world map separated by countries is displayed with the capacity to zoom in to take a more detailed look from which particular regions you are receiving visitors. If you click in the U.S. you can hover the cursor of the mouse over any state and a textbox will pop up with the frequency of visitors. Using the Map Overlay tool you may be able to identify if you need to translate the contents of your website to a specific language, for instance if you are receiving many visitors from non-speaking English countries or communities.

To learn more about map overlay, view Google Analytics in 60 Seconds: Location Targeting on YouTube

Rad Resource – Google Motion charts

Motion charts allows you for instance to identify keywords that people have used to find your website. Keywords can be displayed as dynamic charts using bubbles or bars. A bubble chart may describe the average number of pages per visit using a specific keyboard. What is nice about the motion chart is that allows you to see changes in the use of keywords over time, which may indicate some trends that people are following influenced by a professional forum discussion, participation in events, or particular interests brought up by your followers. As people change their interests and ideas, this is a great information tool for you to adjust the contents of your website according to the needs of your visitors.

To learn more about motion charts, view Motion Charts in Google Analytics on YouTube

The American Evaluation Association is celebrating Data Visualization and Reporting Week with our colleagues in the new DVR AEA Topical Interest Group. The contributions all this week to aea365 come from our DVR members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DVR resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · · ·

Older posts >>

Archives

To top