We’re Jean King and Frances Lawrenz (University of Minnesota) and Elizabeth Kunz Kollmann (Museum of Science, Boston), members of a research team studying the use of concepts from complexity theory to understand evaluation capacity building (ECB) in networks.
We purposefully designed the Complex Adaptive Systems as a Model for Network Evaluations (CASNET) case study research to build on insider and outsider perspectives. The project has five PIs: two outsiders from the University of Minnesota who were not as involved in the network being studied prior to this study; and three insiders, one each from the museums that led the network’s evaluation for over a decade (Museum of Science, Boston; Science Museum of Minnesota; and Oregon Museum of Science and Industry).
Lessons Learned:
Outsiders were helpful because
- They played the role of thinking partner/critical friend while bringing extensive theoretical knowledge about systems and ECB.
- They provided fresh, non-participant perspectives on the network’s functioning and helped extend the interpretation of information gathered to other networks and contexts.
Insiders were helpful because
- They knew the history of the network, including its complex structure and political context and could easily provide explanations of how things happened.
- They had easy access to network participants and existing data, which was critical to obtaining data about the ECB processes CASNET was studying, including observing internal network meetings and attending national network meetings, using existing network evaluation data, and asking network participants to engage in in-depth interviews.
Having both perspectives was helpful because
- The outsider and insider perspectives allowed us to develop an in-depth case study. Insiders provided information about the workings of the network on an on-going basis, adding to the validity of results, while outsiders provided an “objective” and field-based perspective.
- Creating workgroups including both insiders and outsiders meant that two perspectives were constantly present and occasionally in tension. We believe this led to better outcomes.
Hot Tips:
- Accept the fact that teamwork (especially across different institutions) requires extended timelines.
- Work scheduling was individualized. People worked at their own pace on tasks that matched their skills. However, this independence resulted in longer than anticipated timelines.
- Decision making was a group affair. Everyone worked hard to obtain consensus on all decisions. This slowed progress, but allowed everyone—insiders and outsiders–to be an integral part of the project.
- Structure more opportunities for communication than you imagine are needed. CASNET work taught us you can never communicate too much. Over three years, we had biweekly telephone meetings as well as multiple face-to-face and subgroup meetings and never once felt we were over-communicating.
- Be ready to compromise. The different perspectives of team members owing in some cases to their positions within and outside of the network resulted regularly in the need to accept another’s perspective and compromise.
The American Evaluation Association is celebrating Complex Adaptive Systems as a Model for Network Evaluations (CASNET) week. The contributions all this week to aea365 come from members of the CASNET research team. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.