AEA365 | A Tip-a-Day by and for Evaluators

We are Jenny McCullough Cosgrove, Nicole Huggett, and Deven Wisner, the 2017-2018 conference planning committee for the AEA Arizona Local Affiliate, the Arizona Evaluation Network (AZENet). This year, we built off of the momentum from the inspiring #Eval17 AEA conference, to bring focus and meaningful attention to inclusion and equity in evaluative practice at our annual Arizona Evaluation conference.

Cool Tricks: Provide a Space for Evaluators to Practice

We know participatory evaluation can be a powerful tool in advancing equity by explicitly including underrepresented stakeholder voice. Given this, the conference planning committee has worked with our keynote speaker Dr. Mia Luluquisen, Deputy Director of Community Assessment Planning and Education at Alameda County Public Health Department, to build an evaluation event that incorporates an active experience in participatory evaluation. Specifically, an evaluation of the conference will be used as an introduction to this topic.

Hot Tips: Purposefully Build Inclusion and Safety into the Event

  • Choose an event location that will be accessible to all abilities.
  • Design event products and communications so they are as usable by as many participants as possible.
  • Define and use an inclusive and just vocabulary in promotion of the event and during the event.
  • Add activities that focus on experiencing deep empathy.
  • Establish ground rules for active listening; encourage all participants to engage and listen.
  • Support critical reasoning and safety in participants by asking for quiet reflection before sharing ideas.
  • Do not assume that marginalized people have the responsibility to educate evaluators on equity issues. Be mindful of asking underrepresented peoples to teach or explain their needs or experience at your event. Marginalized people are often burdened with the expectation to be the teachers in matters of justice and equity issues.

Rad Resources:

Intrigued and want to learn (or experience) more? Check out the Arizona Evaluation Network’s 2018 From Learning to Practice: Using Evaluation to Address Social Equity conference taking place this April in Tucson, AZ.

The Annie E. Casey Foundation provides seven steps to embed equity and inclusion in a program or organization in the Race Equity and Inclusion Action Guide.

Racial Equity Tools provide some wonderful resources for evaluators to learn more about the fundamentals of racial inequity, as well as useful tools and guides to support learning.

Learn more about disability inclusion strategies from the Centers for Disease Control and Prevention.

Reflect on your strategies for gender inclusion with this guide from the University of Pittsburgh.

The American Evaluation Association is celebrating Arizona Evaluation Network (AZENet) Affiliate Week. The contributions all this week to aea365 come from our AZENet members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Jeneen R. Garcia

Jeneen R. Garcia

My name is Jeneen R. Garcia. I’ve been a full-time evaluator at the Independent Evaluation Office of the Global Environment Facility (GEF IEO) for the last seven years. The GEF is the largest multilateral funder of environmental programs worldwide. Because the programs we evaluate almost all take place in complex social-ecological systems, we constantly need to seek out new methods for dealing with complexity.

One of the methods we’ve used is Social Network Analysis (SNA). In one evaluation, we wanted to assess the role of the GEF in increasing interactions among environmental actors at the regional level. Two things made this system complex:

1) the work of these many actors intersected, but they had no hierarchical structure, and

2) interventions took place at multiple scales, which ultimately shaped interactions at the regional scale.

It’s hard to keep track of what everyone says they’re doing and who they’re doing it with. By mapping the relationships among actors, SNA allowed us to see how well-connected the actors in the region are, and which ones are key to keeping the network well-connected.

Because it was an impact evaluation, we also needed some sort of counterfactual to compare our observations with. The big problem was, it is practically impossible to “randomly select” a region that is comparable to any other, much less find a high enough number of them to ensure statistical robustness. In this case, we were looking at the South China Sea, a region with several territorial conflicts, and which GEF has funded for > 20 years. How could we find a region to compare with that?

Hot Tips

  • Instead of looking outward, we created a scenario of the same region without GEF’s presence. We did this by redoing the SNA with the same set of actors except the GEF. The result was, without GEF support, some actors that were important at the country level became disconnected from the regional discussions.
 SNA diagram with and without GEF

(click for larger image)

  • We did not rely on this analysis alone to assess the impact of GEF funding in the region. We triangulated it with field visits, interviews at multiple scales, document reviews, environmental monitoring data, global databases, and satellite images, among others. A wide range of evidence sources and methods for analysis is your best defense against data gaps in complex systems!

Rad Resources:

To find out more about which SNA measures were used to come up with our findings, you can check out this paper that I wrote up on the analysis.

You can also see how this analysis fits in with the larger impact evaluation by reading the full report.

For more on the basics on the basics of SNA and how it can be used in evaluation, you can explore this Prezi I made. It includes links to evaluations, software, and other resources related to SNA. (CAVEAT: I delivered that presentation to a Spanish-speaking audience and haven’t translated it yet. My apologies to the non-Spanish speakers!)

The American Evaluation Association is celebrating Social Network Analysis TIG Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Rebecca Swann-Jackson, a Senior Research Associate at the Center for Research and Evaluation on Education and Human Services at Montclair State University.  I currently manage evaluation projects focused on teacher preparation and development, and educational programs and community-based initiatives serving urban children and families.  In the evaluation of an urban teacher residency program, I recently used social network analysis (SNA) to examine the relationship between support for novice teachers and retention (i.e., staying in their schools and/or the profession).

Social network analysis is an innovative method used to understand relationships. As relational models, networks show both structure (who and what) and process (how and why) at the same time. Further, you can obtain a more complete picture by combining quantitative (outsider view) and qualitative measures (insider view) of the structure and process.

SNA diagram

Hot Tips: These tips are especially relevant for those who want to try out mixing quantitative SNA with qualitative methods.

The network survey will help to construct the ‘who and what’ relational network. To use the network for evaluation purposes, you also might consider using qualitative methods to investigate ‘how and why’ wonderings. Interrogate, or question, your models; what do you want to know? Ask questions of the relationships and connections you see (and don’t see!).

In the case of the evaluation of the urban teacher residency program, I was curious about:

  1. How does each supporter do their job?
  2. Why do novice teachers reach out to these people for support?
  3. Why do novice teachers reach out to these people for these specific types support?

Tip 1: Have a data party to engage respondents in interpretation and questioning: Reconvene your survey respondents. Distribute copies of the network model with the identifiers removed. Have them identify the questions they have about the model. Ask participants which node they think represents them and ask them to explain their decision-making.

Tip 2: Investigate questions through qualitative inquiry with key nodes. In my evaluation, I used focus groups to further understand the nature of key nodes’ roles. I interviewed the key nodes to learn more about their day-to-day operations.

Lessons Learned: Combining SNA with qualitative methods provided a more holistic understanding of the relationship between support and retention. Learning how people perceived the network and the content and meaning of ties between individuals was essential to understanding network patterns as well as evaluating program implementation and outcomes.

Rad Resources:

Nvivo by QSR – Enabling Qualitative Social Network Analysis https://youtu.be/8cUBQSWgGqg

Robert Wood Johnson Foundation – Using Social Network Analysis in Evaluation https://www.rwjf.org/en/library/research/2013/12/using-social-network-analysis-in-evaluation.html 

The American Evaluation Association is celebrating Social Network Analysis TIG Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Bethany Laursen

Bethany Laursen

Hello, everyone! I’m Bethany Laursen, principal consultant at Laursen Evaluation & Design, LLC and doctoral student at Michigan State University. I love sense making tools, don’t you? I need help untangling complex data into meaningful findings! Social network analysis (SNA) is one of those tools, and it can do a lot more than its name indicates if you know how to hack it.

SNA is fundamentally network analysis, and you can study almost anything as a network. In fact, if you’re a systems thinker like I am, you probably do this already!

Hot Tip: All you need to hack SNA is at least one set of nodes and one set of edges. Stuck? A few inspiring questions include: What is flowing in my network? What do I care about? What is easy to measure?

Here are some basic examples:

Nodes Edges
Bus stops Bus routes
Grants Shared objectives
Land preserves Wildlife migrations
Accounts Fund transfers
Activities Causes

 

Level 2 hacking adds more edges to make a multiplex graph. For example, we might track shared personnel as well as shared objectives among grants. Level 3 hacks add another set of nodes to create 2-mode networks, such as bus stops with ATMs within one block. Combining levels 2 and 3 gets you to level 4—a multiplex, two-mode network (!). There are more secret levels to discover if you create new nodes and edges out of your original ones using the analytic transformations available in SNA software.

For example, I once turned a simple information-exchange network into a two-mode expert-expertise network, and then—through a co-affiliation transformation in UCINET—I ended up with an awesome group concept map of everyone’s shared expertise, where the nodes were expertise types and the edges were people recognized as those experts. How cool is that?

Figure 1: An expertise network made of areas of expertise connected by people who have those expertises. From Laursen 2013.

Figure 1: An expertise network made of areas of expertise connected by people who have those expertises. From Laursen 2013.

Lesson Learned: You can make intangible, complex constructs visible and interpretable by re-purposing SNA.

Lesson Learned: It’s fun to play with the possibilities of SNA, but in the end, you need to have a purpose for the information you generate. Having a good question to answer is half the secret of sense making tools.

 

Rad Resources: Here are some methods and tools that re-purpose SNA:

The American Evaluation Association is celebrating Social Network Analysis TIG Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Sophia Guevara, MLIS, MPA.  I am a co-chair for the Social Network Analysis Topical Interest Group at the American Evaluation Association.   My co-writer is Simon Geletta, who was a past program chair of the SNA TIG. Simon is a professor of public health at the Des Moines University.

In this post we would like to introduce a software called “ORA”.  ORA is an extremely flexible network analysis tool that is ideal for creating, manipulating and analyzing networks and network structure from data that are stored in a number of different ways and formats (e.g., as a set of relational tables stored in a database or in a spreadsheet, as an n-dimensional matrix etc.) It allows visual as well as statistical analysis capabilities on both static social networks and dynamic social networks that can vary over time and/or space.

ORA is versatile, as it is a “multi-platform” toolkit that can operate either in stand-alone mode, or as a service “plug-in” within a web architecture. With both a GUI version and batch mode version of ORA, it is noteworthy to mention that the batch mode version has been used with networks with 106 nodes.  ORA supports high dimensional network data (or “meta-network” data), including data that represent spatiotemporally dynamic network structure. Hence, while most SNA tools are capable of mapping single-mode or two-mode networks, ORA can handle n-mode networks – this makes it ideal for measuring and understanding network changes over time or through space.

A second powerful feature is its ability to visualize geo-spatial networks. The ESRI proprietary geographic “shape” file can be used together with network data to visualize relationships between entities over geographic space. Further, ORA outputs can also be export to Google Earth, or to KML files, thus enabling interoperation with third-party tools.

Finally, ORA is interoperable with a number of other SNA tools such as Pajek and UCINET. Further, its output can be consumed by a wide range of applications because they can be made to conform to CSV, TSV, XML, JSON and similar standards.

According to the Center for Computational Analysis of Social and Organizational Systems (CASOS) website, there is ORA-LITE which is limited to 2,000 nodes and a Pro version with no node limit available at Netanomics.com.  The Netanomics.com site invites visitors to access an article published in The Economist in 2015 that mentions the use of this software.

Rad Resource: ORA Google Group https://groups.google.com/forum/#!forum/ora-google-group

This Google Group provides information for those interested in “network science and network science tools”.  You can find more information on the page about training and purchasing tools.

The American Evaluation Association is celebrating Social Network Analysis TIG Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Rebecca Woodland and I teach Introduction Social Network Analysis as an AEA pre-conference workshop and at UMass Amherst. Those who enroll in these courses come from all walks of the evaluation field and I feel quite fortunate to have the opportunity to learn alongside so many talented professionals.

What I’ve learned.

Whether people seek to conduct evaluation in health and medicine, education, environmental non-profits, or in any number of other settings and sectors, there are some common misconceptions that seem to exist. I too held many of these misconceptions when I first ventured into SNA for program evaluation.

Myth #1 – SNA is about Facebook, Twitter, etc. and other social networking sites.

A lot of people think that the “N” in SNA refers to “networking.” The reality is that there is no “ing” in SNA. Yes, online social networking can be analyzed using SNA, but SNA is not about the study of social networking per se. In the context of evaluation, SNA is most often about examining relationships (ties) between actors (people/organizations) and how a program’s “network” may enable or inhibit actor access to important resources.

Myth #2 – SNA is only about ties between people.

Perhaps because SNA has the word “social” in its title – it is widely assumed that SNA is exclusively about people. Evaluators are rightly concerned about program effects for individuals, but SNA is also a sophisticated way to examine ties between program resources, inputs, outputs, and outcomes. With SNA, we might examine ties between nonprofits and grant-making organizations, program activities and geographic regions, or policies and governmental groups, just to name a few.  An almost unlimited number of relationships between human AND/OR non-human actors can be examined through SNA.

Myth #3 –SNA is all about creating those cool pictures.

It is true that the production of colorful and dynamic sociograms is one of the most powerful aspects of SNA! However, effective SNA does not have to entail the creation of any pictures. SNA is predicated on matrix algebra – sociograms visually depict results of mathematical computations. Sometimes it is more important, accurate, and useful to tell the story of program cohesion and actor centrality using quantitative network measures and descriptive and inferential statistics.

Rad Resources:

If you find SNA intriguing – check out the following resources. Each text addresses SNA misconceptions and may help you to further incorporate this powerful approach into your evaluation practice.

  • SNA: Methods and Applications by Wasserman & Faust
  • SNA: History, Theory and Methodology by Prell
  • Analyzing Social Networks by Borgatti, Everett, & Johnson
  • Social Network Analysis by Scott
  • SNA: Methods and Examples by Yang, Keller & Zhang
  • The SAGE Handbook of SNA

The American Evaluation Association is celebrating Social Network Analysis TIG Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Maryann Maxwell Durland

Maryann Maxwell Durland

I am Maryann Maxwell Durland and I am part of the leadership team of the Social Network Analysis (SNA) TIG, along with Rebecca Woodland, and Sophia Guevara. One important distinction between our TIG and many others is that we are not a primary TIG for many members, but rather a “secondary” choice. Primary choices tend to be the area or field where we work or a specific interest such as the Disaster & Emergency Management Evaluation TIG. A secondary choice is of interest to an evaluator but generally not at the level of attending the business meeting or participating in TIG activities. Our leadership has been addressing this pattern and working to encourage evaluators to become more engaged in their secondary TIG’s, as well as providing resources that will address learning and professional development needs better. To inform this goal, in December 2017, we sent a survey to our members (300+) with a response rate of 12%.

Overall Results indicate that:

  • Half of the 38 members who responded are at the novice and beginner level; Another 13% rate themselves as experts
  • Based on respondents ranking of their confidence level with SNA concepts, we have a mix of members some who are very confident about concepts and a larger percentage who are not confident or have a little confidence.
  • Very few of respondents indicated confidence with software on a three point scale 11% with Gephi, 11% with R and 14% with UCINET. Respondents also listed 11 other software they use.
  • Over 70% of respondents have used SNA in describing a program as part of an evaluation and measuring/visualizing program outputs. Another 68% have used it in creating a final evaluation report, and 55% in engaging stakeholders.
  • The majority of respondents have not read common texts related to SNA concepts and several listed other texts they have read.

In the chart below are the specific concepts we asked about. Based on the survey results, the leadership team thought that we would address, from not confident to the very confident levels, the concepts that are foundational to doing SNA. All of our posts this week will discuss a variety of concepts and uses for SNA. I have started by illustrating how our TIG is addressing how we can be useful to our members. In addition, I’ll provide under Rad Resource, a short description of a classic book for first getting a sense of SNA.

Rad Resource:

John Scott’s  Social Network Analysis (3ed) gives a clear and readable overview of Social Network Analysis. Scott provides key definitions and an historical perspective, as well as covering key conceptual concepts. He includes brief examples in each chapter to further illustrate applications.

The American Evaluation Association is celebrating Social Network Analysis TIG Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi my name is Jayne Corso and I am the Community Manager for AEA. Videos are emerging as the most effective way to reach your followers on Facebook.  Here are a few tips for posting videos that are effective on Facebook.

Hot Tip: Avoid Long Videos

Videos should be short and to the point. The attention span of Facebook users is usually less than a minute. Use this time wisely and identify your topic early in the video to gain the interest of your followers.

Hot Tip: Uploading to Facebook

Facebook often favors videos that are posted directly to their platform. Direct uploaded videos will out preform any photo, link, or even external reference to a video.

Hot Tip: YouTube  

If you decide to house your videos on YouTube, here are a few tips to keep in mind.

  • Include a description on your video
  • Create a short title that sums up the topic of your video
  • Use keywords that relate to the topic of your video
  • Add a date and location to your video for reference

I hope these simple video tips are helpful!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi everyone! Carolyn Fisher, Martina Todaro, and Leah Zallman of the Institute for Community Health here, with the second installment in our epic tale of survey design.

When last we left our heroes, we had arrived at this question to measure vulnerability due to low income, and were about to pilot it with our client’s participants.

Do you get ANY of the following benefits? SNAP/Food stamps, WIC, SSI, SSDI, TANF, Housing Assistance, Medicaid

  • Yes
  • No
  • Unsure
  • I don’t want to answer

Cool Trick: Pilot your surveys! Have volunteers fill out the survey, read each question back, and explain why they gave each answer. We always learn when we do this!

In this case, our piloting gave us an unexpected result: our client and their participants felt that this question wouldn’t work due to the stigma around public benefits — stigma would prevent people from answering. Further, they felt some people taking the survey could be undocumented immigrants, who, even when eligible for benefits, are often reluctant to apply.

We weren’t quite ready to let go of this question, though. In trying to tweak it to address these issues, we came up with:

Just based on your income, are you or would you be eligible for any of the following benefits? Food stamps, cash assistance, housing assistance, or [state specific Medicaid name]?

  • Yes
  • No
  • I don’t know
  • I don’t want to answer

I hope everybody reading this is cringing! Perhaps the less said about this version, the better.

Soon coming to our senses, we decided to re-focus on the actual domain we were trying to assess: vulnerability. The following is based on validated measures of social determinants of health.

In the past 12 months, have any of the following been true? (check all that apply)

  • You worried whether your food would run out before you got money to buy more
  • You worried about losing housing or were homeless
  • The electric, gas, water or oil company threatened to shut off services in your home
  • None of the above
  • I don’t want to answer

Rad Resource: Health Leads USA has published a toolkit with validated and unvalidated measures for domains of social needs and social determinants of health.

This final question both:

-Measures the actual construct we needed to know about (vulnerability)

-Is responsive to the contexts where it is fielded

We hope that hearing about our journey with this question was helpful to you in your own work — happy voyaging!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi there, fellow evaluators! We are Carolyn Fisher, Martina Todaro, and Leah Zallman, of the Institute for Community Health. ICH is a nonprofit consulting organization that specializes in participatory evaluation, applied research, and strategic planning.  We help health care systems, governmental agencies, and community-based organizations improve services and create meaningful impact. In a two-part blog entry, we’re going to tell you about a survey design problem we encountered that was unexpectedly tricky.

Once upon a time, we were creating a survey to help a client understand if their program was reaching vulnerable members of low income communities.

At first, we assumed we would define “low-income” as “below the federal poverty level (FPL)”. We thought we’d ask:

What is your household income? ______________

To determine whether a household falls above or below the FPL, which varies based on household size, we’d also need to ask:

How many people in your household?   ______

However, we found a number of problems with this set of questions.

  • Household income may be difficult to calculate for some households. Do you know the income of everybody you live with? Do you share all expenses? Do you count the income of your adult child? Do you count the income of someone who only lives there part of the year?
  • Household size is also a tricky thing to ask for some households! In particular, people may not know how to calculate their household size if they have members who do not live there permanently, or with people who contribute to the household’s income but do not live there. This is most common among the economically vulnerable households we expected to be identifying.
  • Poverty and vulnerability are relative to the cost of living. A household at the FPL is better off in rural Alabama than in New York City, for example.
  • In our experience, many respondents skip survey questions about income. This could be for the reasons above, but also because of taboos about money and financial vulnerability.
  • Finally, we didn’t need this much information!

Hot Tip:  In survey research, limit the questions you’re asking to what you really need to know. Here, we only needed to know whether a respondent was vulnerable due to their low income. This is fundamentally a Yes/No question.

Our next idea was to use a proxy measure for low income, such as:

Do you get ANY of the following benefits? SNAP/Food stamps, WIC, SSI, SSDI, TANF, Housing Assistance, Medicaid

  • Yes
  • No
  • Unsure
  • I don’t want to answer

This was a less-problematic question than our first attempt, because it was easier for the respondents to calculate. However, it was only our second stop on a journey that wasn’t over yet…

To be continued!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top