ACM TIG Week: Collaborating with Practitioners to Create Relevant Instruments to Measure Library Staff Capacity for Informal Learning Activities by Patricia Montaño, Megan Littrell, Anne Gold, Claire Ratcliffe Adams, & Brooks Mitchell

We are Patricia Montaño, Megan Littrell, and Anne Gold from CIRES Education & Outreach at the University of Colorado, Boulder, and Claire Ratcliffe Adams and Brooks Mitchell from the National Center for Interactive Learning (NCIL) at the Space Science Institute in Boulder, Colorado. We collaborate on the We are Water project to design informal Science, Technology, Engineering, Art, and Math (STEAM) experiences for patrons of tribal and rural public libraries in the Four Corners region. Merging the expertise of practitioners, researchers and evaluators ensures our work is  informed by and responsive to the contexts we are studying.

A collaborative process includes making sure the development of research and evaluation instruments is aligned with informal education design and implementation. To achieve this, we created a partnership between practitioners, trainers, and educational researchers seen in these three groups: 1) Library staff who host the We are Water exhibition and facilitate STEAM activities during hosting, 2) the National Center for Interactive Learning (NCIL) who train library staff in STEAM activities and provide resources and support through the STAR Library Network, and 3) CIRES Education & Outreach who designs the educational research and evaluation.

By bringing practitioners, trainers, researchers, and evaluators together from the start, and through each step of the project, we were able to incorporate multiple perspectives into the development of instruments designed to measure library staff capacity to engage patrons in STEAM. If one group had not been at the table, the development of instruments would not have been as valid and reflective of the participants nor as well tested.

Through conversations, we realized that we had slightly different understandings about the context of rural library settings, and context is everything. Hearing from and being advised by rural library staff quickly corrected misunderstandings. For example, rural libraries are super-stars at serving large geographic areas with a small staff ranging from 2 to 4 people, and each staff member is often responsible for all library functions. This is different from larger libraries, where responsibilities are divided among more staff members. Moreover, the pandemic put unique stressors onto rural library work. Library staff were expected to provide public and social services beyond their usual job descriptions. Understanding the context of rural libraries and the experiences of their staff was key in creating relevant training modules and survey questions. Library staff provided us with pivotal insights that affected training formats and ensured greater participation in research and evaluation.

We also learned from library staff that we had different definitions about what STEAM programs looked like and what “counted” as a program. We broadened our definition with the more inclusive phrase, “learning experiences.” In doing so, we could encompass all the efforts of library staff, such as topical book displays and StoryTime.

Lessons Learned

Even if you have experience with a community organization like a library, never assume your experience gives the full picture. Always ask questions, always listen, and expand your understanding.

In a group with different perspectives, it’s unlikely everyone has the same definitions for words and phrases you commonly use in your jobs. It’s always a good practice to ask everyone to share, describe, and give examples to illustrate what those words and phrases mean.

Rad Resources

  • Library Research Service has information and tools on research and evaluation in libraries.
  • STARNet has a wealth of STEM resources for library staff.
  • InformalScience.org has free evaluation and research reports on informal STEM education in libraries and on library staff capacity.

We honor and acknowledge that the University of Colorado’s four campuses are on the traditional territories and ancestral homelands of the Cheyenne, Arapaho, Ute, Apache, Comanche, Kiowa, Lakota, Pueblo and Shoshone Nations.


The American Evaluation Association is celebrating Arts, Culture, and Museums (ACM) TIG Week. The contributions all week come from ACM members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.