Hi, I am Liudmila Mikhailova, an evaluation practitioner with 28 years of experience designing, managing, and evaluating U.S. government foreign assistance/diplomacy programs with fieldwork in 25 countries. I also teach two George Washington University’s International Education Program courses: International Program Evaluation and International Program Design.
The last decade has witnessed a significant effort in the U.S. government (USG) towards strengthening requirements for U.S. foreign assistance program evaluation. One example is the Presidential Memo (2021) on scientific integrity and evidence-based policymaking which stated that evidence-building plans and annual evaluation plans of federal agencies “shall include a broad set of methodological approaches for the evidence-based and iterative development and the equitable delivery of policies, programs, and agency operations.”
To comply with USG requirements, non-profit organizations and private evaluation companies apply methodological rigor by designing and aligning evidence-based evaluations with the Guiding Principles for Evaluators and the AEA Public Statement on Cultural Competence In Evaluation. The latter document states: “Evaluations cannot be culture free. Culture shapes the ways in which evaluation questions are conceptualized, which in turn influence what data are collected, how the data will be collected and analyzed, and how data are interpreted.”
Even though evaluators thrive to increasingly focus on culturally responsive evaluations to accurately reflect life experiences of program participants in different socio-cultural contexts, often the importance of culture in program implementation and evaluation is underestimated.
Working with Diverse Cultures
- Use cross-cultural expertise to advocate for cross-cultural sensitivity in international program implementation and evaluation. For example, “losing face” is a big shame in collectivistic, hierarchical cultures, so talking to grantees about what did not work should be done in a culturally appropriate way.
- Educate yourself and others on cross-cultural competence from books and colleagues! We have great diversity in our teams of programmers and evaluators. Sharing knowledge with SMEs and colleagues via webinars or presentations will contribute to overall cross-cultural awareness raising and help properly design and executive culturally responsive evaluations.
- Pilot all evaluation instruments (we all do!) Listening to voices on the ground and learning what constitutes success from “their perspectives” and which outcomes they associate with success is critical. In Ukraine, e.g., during weeklong in-depth interviews and focus groups with selected institutions, we requested PIs and their teams to review survey questions for cultural accuracy. We applied a similar approach during our fieldwork in the Philippines, Indonesia, Mexico City and Uganda. The results were rewarding and allowed for finalizing a culturally responsive survey and yielding great data.
- Be a strong advocate for sharing evaluation results with the funder, colleagues, and other stakeholders. Effective ways include debriefings, presentations, and seminars. Engaging in-country partners and grantees in reviewing evaluation findings helps not only build continuing trust but also lays a long-lasting foundation for sustainable collaboration and engagement.
Some examples of methods I found useful and contributing to culturally responsive and equitable approaches in evaluation practice:
- Mixed methods (Cresswell J.W. & Plano Clark V. L. (2011). Designing and Conducting Mixed Methods Research, Sage) – is a recognized approach in non-experimental design.
- Geert Hofstede’s dimensions of culture helps understand the pace of change in a given culture where change occurs.
- Ricardo Wilson-Grau Outcome Harvesting method is an effective tool to engage direct beneficiaries in formulating outcome descriptions and identifying additional outcomes that clarify the “meaning of change” in complex socio-cultural systems.
- Rick Davis Most Significant Change (MSC) technique is effective because it allows for engaging stakeholders in a discussion what change should be highlighted while analyzing data on impact and outcomes.
- UNDP Discussion Paper: Innovations in Monitoring & Evaluating Results – I found this source useful to see what methods and tools are considered innovative and how they can be used.
I welcome everyone to contribute to the question we ask ourselves: What are best practices, methods, and tools you utilize to ensure culturally responsive and equitable evaluations?
The American Evaluation Association is hosting International and Cross-Cultural (ICCE) TIG Week with our colleagues in the International and Cross-Cultural Topical Interest Group. The contributions all this week to AEA365 come from our ICCE TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
3 thoughts on “ICCE TIG Week: Culturally Responsive and Equitable Approaches in Evaluation Practice by Liudmila Mikhailova”
Thank you for this insight,
I am imagining that evaluation practices, effects and uses will vary within cultures. What is perhaps an intrusive methodology in one domain my be seen as ineffective in another. I think that your hot tips section is a great summarization of effective approaches to dealing with diverse communities. This is true within micro cultures as well, not just other countries. I think the advice laid out in your article can be transposed to all evaluations to ensure that a responsive approach is taken to appreciate and adapt to diverse needs.
I would agree with you that this could easily be an overlooked aspect of the profession. Thank you for bringing attention to it and sharing your experience!
I enjoyed reading your blog as it discusses the importance of culturally responsive and equitable approaches to evaluation.
You clearly stated how “evaluations cannot be culture free”. I immediately started thinking about the program I am working in at the moment and started thinking about how that statement connects to my cliental. I think you have made an excellent point, where programs are not necessarily built for culture-based perspectives.
An example is the education system. It is very western and valued in most parts of the world. I think we are moving in the right direction now with incorporating in our BC curriculum Indigenous Knowledge and culture however, this has been missed for many years.
I then connected your blog to overall assessment. Not in a program setting but in a classroom setting. How can I incorporate different cultures into my evaluation of student learning? What or will there be change and impact? If so, what will that look like?
I really enjoyed reading your “hot tips” and will use those moving forward in my program evaluation for the end of the year.
Thank you for sharing your post. I look forward to hearing from you,
Thank you, this is great!
I am lucky enough to be an internal evaluator embedded within a project team from the early stages of a project. I am hoping to refine our evaluation indicators to reflect the values and preferences of the end-users with whom we are consulting. How can I inquire deeper than “what would success look like to you”? Could anyone recommend approaches that have worked well in focus group type settings?