This week, we celebrate the theme of cluster, multi-site, and multi-level evaluation with its topical interest group.
This blog was originally posted on May 27, 2012.
My name is Gary Resnick and I am the Director of Research at Harder+Company Community Research, a California-based consulting firm. My background combines program evaluation with child development research, and I have an interest in system theory and networks.
Harder+Company has been involved evaluating First 5 programs in a number of California counties. First 5 arose from 1998 Proposition 10, adding a tax on tobacco products with funds distributed to counties to fund local programs that improve services for children birth to 5 and their families. An important goal of First 5 funding is to act as a catalyst for change in each county’s systems of care. To measure system change, we focused on inter-agency coordination and collaboration. Increases in coordination and collaboration would indicate that agencies are better able to share resources and clients, reduce redundancies and service gaps, and increase efficiency.
Rad Resource
The Levels of Collaboration Scale assesses collaboration, has excellent psychometric properties and can be administered in web-based surveys to agency respondents. To see it in action, check out this article in the American Journal of Evaluation. Originally a 5-point Likert scale, we combined the two highest scale points creating a 4-point scale to make it easier for respondents.
Hot Tip
Start by defining the network member agencies using objective, clear, and unbiased criteria. Later, you can expand the network by asking respondents to nominate up to three additional agencies with whom they interact.
Hot Tip
Select at least two respondents from each organization, three is better, from different levels of the organization, administrators and managers as well as direct line staff.
Lesson Learned
It is important to have complete, reciprocal ratings for each agency (even if not from all respondents). If you have too much missing data at the agency level, consider excluding the agency from the network.
Hot Tip
Use Netdraw, a Windows freeware program, to produce two-dimensional network maps from agency-level Collaboration Scale ratings. See our maps here. The maps identify agencies most involved with other agencies at the center of the map (key players) and those least involved, at the periphery of the network. Add attributes of agencies (e.g. geographic region served) to map subgroups of your network.
Hot Tip
Produce two sets of maps, one with no agency labels for public reporting, and another with agency labels, for internal discussions with clients and agencies. Convene a meeting with the agency respondents and show them the maps with agency labels, to help them understand where they stand in the network and to foster collaboration.
Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Great article! But I think many of the original links are broken. Do you have any updated links? Thank you!
Thanks for pointing that out, Rachel. We’ve updated the links in the post.