We, Courtney Stone, Ayesha Boyce, and Audrey Amrein-Beardsley are a graduate student, associate professor, and a professor, respectively, at the Mary Lou Fulton Teachers College at Arizona State University. We work in the Department of Educational Leadership and Innovation, with specializations in Educational Policy and Evaluation, and collective research interests surrounding culture and climate study constructs, survey research, and research and evaluation methods, underscoring challenges involved with researching diverse groups. We have broad experience evaluating a variety of US National Science Foundation (US NSF), US Department of Education, and Department of Defense projects.
Culture and climate studies offer a way for evaluators to explicitly attend to issues of access, equity, and inclusion while assessing political, social, and cultural facets of programs and organizations. With such potential comes a multitude of challenges such as identifying measurable and relevant constructs, designing survey and other measurement instruments, collecting data, and conducting data analyses using multiple methods, especially in Science, Technology, Engineering, and Mathematics (STEM) contexts.
What are Culture and Climate Studies?
Researchers conducting culture and climate studies typically aim to assess organizational culture and climate, which are distinct concepts. Organizational culture is more stable than climate and includes long-term values, norms, and behaviors. It can be examined using measures related to leadership, innovation, mission, and supportiveness. Climate is connected to experiences and perceptions as it is more immediate, flexible, and susceptible to current events and personal dynamics than culture. It can be examined using outcome-driven and experiential measures related to fairness, safety, and engagement.
Education research related to climate has shown that educational outcomes are connected to climate aspects such as psychological climate and behavioral climate. What began as reactive measures specifically as a result of racial incidents on college campuses, climate studies or assessments have shifted to a proactive evidence-based approach to identify specific areas for improvement most specifically for racial climate, although studies of other climates from the perspectives of other groups have also been conducted.
Challenges of Culture and Climate Studies
The majority of previous studies, however, have not been conducted with ethnically diverse student populations, with many being conducted in Predominantly White Institutions and a few in Historically Black Colleges and Universities. The limited research on diversity culture and climate focuses on higher education settings and is based on frameworks that lack emphasis on structural, organization, and external influences on such climates, all of which are relevant to consider and engage with in more modern contexts. Furthermore, many culture and climate frameworks have been derived from, developed for, and validated with homogeneous samples of study participants, exclusive of diverse participants with more diverse perspectives and identities.
We conducted an exploratory factor analysis on a culture and climate survey we used across two US NSF programs over the span of two years. Our sample consisted of both faculty and students of diverse backgrounds. We found our instrument to be highly reliable and want to share what we learned from our experience.
Lessons Learned
- Context is Key – Many existing culture and climate surveys are program- or organization-specific which limits generalizability or transferability to different contexts.
- Culture and Climate are Distinct Constructs – Climate can be examined more easily than culture particularly in a short-term program.
- Culture and Climate are Multifaceted Constructs – Outcomes are just one facet of measuring climate and should be considered in relation to other facets of climate (e.g. social, political, culture).
Rad Resources
This blog compares and contrasts culture and climate as concepts.
This systematic review defines other constructs related to culture and climate and reviews psychometric evidence of related measures.
The American Evaluation Association is hosting Quantitative Methods: Theory and Research Evaluation Week. The contributions all this week to AEA365 come from our Quant TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.