I am Russell Cole, a researcher at Mathematica Policy Research. At Mathematica, we have used Social Network Analysis (SNA) in a number of systems change evaluations. The goals of system change interventions are to affect individuals, organizations, and communities, with changes in outcomes occurring at various levels. In our work, we have used SNA to examine relationships among organizations in these types of evaluations.
Rad Resource: A report on how to conduct systems change evaluation by Margaret Hargreaves includes a description of how SNA data can be used to define system boundaries and identify relationships for evaluation purposes (see Evaluating System Change: A Planning Guide).
Hot Tip: When using SNA for systems change evaluation, identify the key organizations involved in the system. If key organizations are missing or excluded from consideration, then the observed network will be incomplete and potentially misleading. Evaluators should develop clear criteria and decision rules to define which organizations to include (and exclude) in the SNA. Creating transparent inclusion rules will define the system boundaries, and the inclusion rule will maintain a consistent representation of the organizations in the system. (See also Stacey Friedman’s AEA365 post this week on defining network boundaries.)
Lesson Learned – The organizations participating in a systems change initiative may change over time. During the planning stages for the systems change initiative, consider how or why organizations might enter or leave the system, and hypothesize what impact these changes might have on inter-organizational relationships or performance. As a result of the systems change initiative, organizations not originally included in the system may become involved and may meet the evaluation’s inclusion rules. Similarly, some organizations originally in the population may dissolve or become obsolete and will become ineligible members of the system. Documenting the entry and exit of organizations into and out of the system change effort is one way to be responsive to the population of interest for the systems change evaluation. As recommended above, population changes should be based on transparent inclusion/exclusion criteria in order to ensure that the observed social structure at each time point is grounded in a consistent definition. (See Evaluating Systems Change Efforts to Support Evidence-based Home Visiting: Concepts and Methods as an example of how a single inclusion rule will be used to characterize the population of interest over time in Mathematica’s Supporting Evidence Based Home Visitation evaluation.)
The American Evaluation Association is celebrating SNA TIG Week with our colleagues in the Social Network Analysis AEA Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members and you can learn more about their work via the SNA TIG sessions at AEA’s annual conference. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.