Hello and Hola. We’re Nnenia Campbell, research assistant at the Natural Hazards Center, University of Colorado Boulder and Saúl Maldonado, research assistant at the ELLISA Project, University of California, Santa Cruz. We are recent graduates of the AEA Graduate Diversity Education Internship (GEDI). One of the GEDI goals is to: “Deepen the evaluation profession’s capacity to work in racially, ethnically and culturally diverse settings.” In the first of this two-part series, we shared a few suggestions for amplifying definitions of diversity. In this second-part, we share a few of the lessons we’ve learned that frame diversity as a measure of validity in culturally responsive evaluation.
Diversity informs validity. Cultural responsiveness is a critical feature of the evaluation process and increases the utility and truthfulness of results (Frierson, Hood & Hughes, 2002). Practicing cultural responsiveness in settings that are racially, ethnically, culturally or otherwise diverse requires considering cultural contexts as technical properties of validity. As evaluators, positioning cultural contexts as validity measures may include review procedures such as the accuracy with which programs serve diverse populations as well as how evaluators select instrumentation and how participants of diverse cultural backgrounds interpret items.
Additionally, cultural responsiveness is co-constructed between evaluators and evaluands. We must seek to identify and communicate the diverse worldviews that are represented in the evaluation process and, more importantly, how diverse beliefs, perspectives, behaviors and values can serve as “resource differences.” Such communications should be continuous, throughout initial, intermediate and final stages of evaluations.
Instrument Selection. When deciding upon which tools and instruments will be used as evaluative measures, it is important to recognize the diversity of our professional training. As much as possible, a range of recommended tools and instruments from multi-disciplinary perspectives should be shared with program participants and organizations.
Item Interpretation. When evaluative measures deemed appropriate are co-selected by evaluators and evaluands, determining the degree of item’s construct-relevance is necessary. Practicing cultural responsiveness requires being attentive to and appreciative of the possibilities for cultural variation. Including participants’ life contexts, prevailing socio-economic conditions, cultural communication and socialization styles in relation to how items are interpreted may reveal the need to, for example, modify measures’ number of items or adapt measurement processes.
Review Procedures. After selecting appropriate instruments and establishing items’ construct-relevance, inclusionary procedures to establish representation of program participants’ and organizations’ diversity is needed. Integrating the perspectives from culturally diverse panels in the interpretation of findings and presentation of results enhances the accuracy and validity of an evaluation.
Regardless of evaluation approach (i.e., participatory, utilization, systems) or method, being attentive and appreciate of diverse cultural contexts enhances both internal and external validity. To reframe diversity as a measure of validity, we recommend the following resources: Frierson, Hood and Hughes’ A guide to conducting culturally responsive evaluation as well as Basterra, Trumbull and Solano-Flores’ book, Cultural validity in assessment: Addressing linguistic and cultural diversity.