Guiding Collective Impact through Evaluation by Teresa McCaffrey

Hello! My name is Teresa McCaffrey and I am the Director of Research and Evaluation at the Educational Partnership Center (EPC) at the University of California, Santa Cruz. The EPC consists of 10 programs, working with over 11,000 underserved middle and high school students each year.

Housing multiple programs with similar goals provided us with an opportunity for collective impact. To that end, we have developed a common agenda, shared measurement systems, and mutually reinforcing activities. Doing so has given us a roadmap to follow, a way to mark our location on that map, and the opportunity for driver (service provider) input.

1. Creating a common agenda.  Our programs share the overarching goal of facilitating educational equity and opportunities for underserved students. Developing an integrated logic model that unified programs in a single logic model template helped staff align mutually reinforcing activities and connect them with short, mid, and long-term outcomes across programs. The integrated logic model is our roadmap.
2. Developing shared measurement systems. Next, we operationalized the integrated logic model by bringing most of our programs onto Lacai, a customized data management system. In doing so, we created a coding scheme that captures how similar services are delivered across programs and started tracking students’ progress toward shared outcomes. Lacai allows us to mark where we are on our journey.
3. Engaging all staff in discussions. Finally, data placemats enable regular conversations and learning within and across EPC college access programs about progress towards logic model outcomes and how to modify activities to achieve them. The placemat analysis offers the venue for driver input and reflection.

Lessons Learned:

• Change is hard. This was especially the case with the shift to Lacai. Even within programs, there were staff who immediately saw the benefits of Lacai (e.g., electronic sign in, real time reports) and having all programs on one database. Others struggled to make the change.
• Allow for program individuality while creating overarching tools. This enables programs to preserve their identity yet work towards unified goals. For instance, in our Lacai coding scheme, we created a matrix of color-coded “objectives” categories (e.g., supporting academics, family services) that align with the activities of the logic model but each program uses different terms for services that address those objectives (e.g., tutoring/homework assistance vs algebra 1 tutoring).
• Share data by program but allow time to talk across programs. Our programs have diverse mandates, funding, and goals, so we use data placemats to discuss data among each program’s staff while also providing time for cross-program discussion.

Rad Resources:

• For more information on collective impact, Kania and Kramer’s 2011 piece in Stanford Social Innovation Review is clear and concise.
• Cooper and Rocha-Ruiz describe the how and why of integrated logic models.
• Pankaj and Welsh provide information about data placemats

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.