Hello AEA 365 Readers! We are Kurt and Aaron, independent evaluation consultants with Effect X LLC. Our experience includes international and cross-cultural evaluations and capacity building projects around the world for organizations such as UNICEF, The World Food Program, The World Bank, Compassion International, Heifer International and The Carter Center.
For those new to the field, the most formidable challenge to international and cross-cultural evaluation might seem to be the language barrier. With experience, however, it becomes clear that language (i.e., knowing the right words) is the tip of the iceberg; the underlying challenge isn’t as much the words, but rather differing cultural perspectives and assumptions that inform the meaning behind the words.
We believe that the first step toward effective cross-cultural evaluation is a clear-eyed recognition of the unique culture we each bring to our work. This is especially important to note for those of us from Western cultures. Psychological literature provides a helpful (but humbling) label for this culture: the acronym WEIRD – Western, Educated, Industrialized, Rich and Democratic. People from WEIRD contexts are “often at the extremes of global distributions…tending to be more individualistic, independent, analytically minded, and trusting strangers” as compared with the rest of the world (Schulz et al., 2018). We evaluators, with our typically higher levels of education and persistent tendencies toward analysis, must contend with our WEIRD roots.
The analytic approach at the foundation of most evaluation design includes ‘operationalizing’ the outputs and outcomes. This step involves distinguishing (or dis-integrating) the program design and human response to the point where causal connections can be examined independent from context. Primary attention is given to the categories to which these disconnected objects belong using rules and formal logic (e.g., rubrics). In contrast, the more holistic and relational cultures of the majority world often see change as a constant phenomenon within a universe of interconnected elements, and causality is understood to be dependent on the entire interconnected field with relatively little use of categories and formal logic (Nisbett et al., 2001 & Li, et al., 2018). In short, learning from the holistic worldviews more common in majority world cultures is an important step toward not just cultural relevance within specific evaluations, but also the broader goal of decolonizing evaluation in general.
One practical step toward a more holistic approach to evaluation is to intentionally integrate the intangible aspects of humanity (e.g., trust, motivation, hope, faith, or love) in evaluation designs and tools. Depending on the evaluation, intangibles can either be addressed as primary contributions to other visible outcomes (e.g., motivation is critical to educational attainment, trust is essential for community development) or could be prioritized outcomes in themselves (e.g., increased hope or agency.) While this might be unfamiliar territory for many evaluators, the resources noted below highlight that there is an interested community and growing range of resources to help get started.
Rad Resources
- Evaluating Intangibles Community of Practice: A vibrant monthly discussion with other evaluators, developing a growing resource database underscores the emerging interest in this topic. Contact learnmore@effectx.com to learn more and join today.
- Journal of MultiDisciplinary Evaluation Article: Intangible Outcomes: The Importance and Current Neglect Within Evaluation Practice
- The Lightwheel evaluation toolkit for measuring holistic change
- Everyday Peace Indicators informed by communities which define for themselves the everyday indicators that they use to measure successful peace in their own communities.
- Peer reviewed journals about psychological measurement: Psychological Assessment, Applied Psychological Measurement, Educational and Psychological Measurement
The American Evaluation Association is hosting International and Cross-Cultural (ICCE) TIG Week with our colleagues in the International and Cross-Cultural Topical Interest Group. The contributions all this week to AEA365 come from our ICCE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.