TAG | diversity
I am Ravan Charles, an evaluation newbie from Omaha, Nebraska. I’m writing about how my personal experience at the 2015 Minnesota Evaluation Studies Institute (MESI) Spring Training will influence the way I do evaluation for the rest of my life.
I have to admit, I was skeptical when I saw the conference theme – ‘Social Justice Amidst Standards and Accountability: The Challenge for Evaluation’ with an emphasis on cultural competence. It is probably important to note that I am a black woman. I grew up in a world where cultural competence was for white people. Cultural competence meant the white lady facilitating my all-black tween girls’ group was able to code-switch fluently, or my sociology professor making sure to call on me whenever we talked about race in class (every day, it was a long semester). My perception was that cultural competency was something that white people were trained to be good at (by other white people).
When I walked into the conference, I instinctively scanned the room for other black and brown faces. I saw one… two… seven…. I lost count. Of the many trainings, conferences, and college classes that I have attended in my life this was the very first where I felt that there was adequate people-of-color representation. I had never realized how important racial diversity is to me until my need for it was satisfied. Once it was, I was able to notice and more deeply appreciate other types of diversity in the rooms. Furthermore, I was able to become more actively engaged and take ownership of the training.
Lesson Learned: When I was able to see myself across the room, in the lunch line, and even at the podium I no longer felt like a spectator. I realized that being culturally competent is about continuously learning from, sharing, and honoring our differences and using that knowledge to create things together. That is just as valuable to me as anyone else. Seeing myself in the room allowed me to see myself as an evaluator, and as someone who wants to be a good one.
I plan to use this experience inform my future work. For me, being culturally competent will mean that I not only strive to interact effectively with clients, I will also work at ensuring that every individual that participates will be able to see their self in every part of the work. My hope is that if every participant feels well-represented throughout they will feel the same sense of ownership and engagement I felt at MESI.
- Cultural Competence in Evaluation: An Overview by Saumitra SenGupta, Rodney Hopson, Melva Thompson-Robinson
- Humberto Reynoso-Vallejo on Cultural Competence and Cultural Humility in Evaluation
The American Evaluation Association is celebrating MESI Spring Training Week. The contributions all this week to aea365 come from evaluators who presented at or attended the Minnesota Evaluation Studies Institute Spring Training. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
My name is Ray Kennard Haynes and I am an Assistant Professor at Indiana University- Bloomington and I have a keen interest domestic racial Diversity in Higher Education (HE). Since the 1970s the United States (U.S.) has attempted to address Diversity by focusing primarily on race and gender through Equal Employment Opportunity (EEO) legislation. This legislation produced some gains; however, those gains have now eroded and are under threat due to legal challenges.
HE institutions in the US have ostensibly embraced Diversity and even claim to manage it. Evidence of this commitment to diversity can be seen in the proliferation of Diversity offices and programs at HE institutions and with the advent of the position of Chief Diversity Officer (CDO). The casual observer could reasonably conclude that Diversity has been achieved in HE. Surely, we see evidence of this reality with the CDO position and ubiquitous Diversity commitment statements. Note too, that the term university can also be construed as: the many and different in one place. Given this meaning and the fact that one in every two U.S. residents will be non-white by the year 2050, Diversity in higher education is a fait accompli. Is HE really diverse with respect domestic racial groups (i.e. African-Americans and Latino-Americans)?
Hot Tips: Research suggests that despite increasing racial diversity, communities and schools are re-segregating to levels representative of the 1960s. In highly selective institutions, diversity has come to mean many things and underrepresented domestic students and faculty are becoming an increasingly smaller part of the Diversity calculus. The evidence suggests HE is becoming less domestically diverse because of the negative co-variation between increases in domestic racial diversity and decreasing access for African-Americans and Latino-Americans to higher education, especially at highly selective schools.
One way for HE to show its commitment to domestic Diversity is to define and evaluate it within the broader construct of DIVERSITY that includes visible and non-visible differences.
Evaluation checklists can be applied to assess domestic diversity deficits and related program implementation thoroughness.
For HE institutions and evaluators who believe that domestic diversity matters, a good place to start is to create Domestic Diversity Evaluation Checklists that assess for both Diversity and Inclusion. These checklists should include dimensions that capture:
- Diversity investment: the budget (investment) associated with domestic racial diversity
- Structural diversity: the numbers of underrepresented domestic students and faculty
- Diversity Climate: decision making and the level of meaningful cross-race interaction and inclusion in shaping the culture and direction of the HE institution
The American Evaluation Association is celebrating Multiethnic Issues in Evaluation (MIE) Week with our colleagues in the MIE Topical Interest Group. The contributions all this week to aea365 come from MIE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello and Hola. We’re Nnenia Campbell, research assistant at the Natural Hazards Center, University of Colorado Boulder and Saúl Maldonado, research assistant at the ELLISA Project, University of California, Santa Cruz. We are recent graduates of the AEA Graduate Diversity Education Internship (GEDI). One of the GEDI goals is to: “Deepen the evaluation profession’s capacity to work in racially, ethnically and culturally diverse settings.” In the first of this two-part series, we shared a few suggestions for amplifying definitions of diversity. In this second-part, we share a few of the lessons we’ve learned that frame diversity as a measure of validity in culturally responsive evaluation.
Diversity informs validity. Cultural responsiveness is a critical feature of the evaluation process and increases the utility and truthfulness of results (Frierson, Hood & Hughes, 2002). Practicing cultural responsiveness in settings that are racially, ethnically, culturally or otherwise diverse requires considering cultural contexts as technical properties of validity. As evaluators, positioning cultural contexts as validity measures may include review procedures such as the accuracy with which programs serve diverse populations as well as how evaluators select instrumentation and how participants of diverse cultural backgrounds interpret items.
Additionally, cultural responsiveness is co-constructed between evaluators and evaluands. We must seek to identify and communicate the diverse worldviews that are represented in the evaluation process and, more importantly, how diverse beliefs, perspectives, behaviors and values can serve as “resource differences.” Such communications should be continuous, throughout initial, intermediate and final stages of evaluations.
Instrument Selection. When deciding upon which tools and instruments will be used as evaluative measures, it is important to recognize the diversity of our professional training. As much as possible, a range of recommended tools and instruments from multi-disciplinary perspectives should be shared with program participants and organizations.
Item Interpretation. When evaluative measures deemed appropriate are co-selected by evaluators and evaluands, determining the degree of item’s construct-relevance is necessary. Practicing cultural responsiveness requires being attentive to and appreciative of the possibilities for cultural variation. Including participants’ life contexts, prevailing socio-economic conditions, cultural communication and socialization styles in relation to how items are interpreted may reveal the need to, for example, modify measures’ number of items or adapt measurement processes.
Review Procedures. After selecting appropriate instruments and establishing items’ construct-relevance, inclusionary procedures to establish representation of program participants’ and organizations’ diversity is needed. Integrating the perspectives from culturally diverse panels in the interpretation of findings and presentation of results enhances the accuracy and validity of an evaluation.
Regardless of evaluation approach (i.e., participatory, utilization, systems) or method, being attentive and appreciate of diverse cultural contexts enhances both internal and external validity. To reframe diversity as a measure of validity, we recommend the following resources: Frierson, Hood and Hughes’ A guide to conducting culturally responsive evaluation as well as Basterra, Trumbull and Solano-Flores’ book, Cultural validity in assessment: Addressing linguistic and cultural diversity.
Hello and Hola. We’re Nnenia Campbell, research assistant at the Natural Hazards Center, University of Colorado Boulder and Saúl Maldonado, research assistant at the ELLISA Project, University of California, Santa Cruz. We are recent graduates of the AEA Graduate Diversity Education Internship (GEDI). One of the GEDI goals is to: “Deepen the evaluation profession’s capacity to work in racially, ethnically and culturally diverse settings.” In this first of a two-part series, we share a few of the lessons we’ve learned that have led to our belief that definitions of diversity require amplification.
Diversity is not just race and ethnicity. Addressing the evaluative needs and interests of programs and organizations requires a consideration of the complex relationship between demographic composition and institutional factors. Program participants and organizations are never homogeneous. For example, factors such as age, gender, sexuality, language, religion, spirituality and national heritage as well as socio-economic and geospatial characteristics are significant traits that also intersect with racial and ethnic factors.
Additionally, it is important for evaluators to consider the relevance of political and/or socio-historical factors that may influence relationships between stakeholders. Paying attention to context offers insights to important underlying dynamics that influence the concerns expressed by key players. As evaluators, it is important to share with our stakeholders, in culturally responsive ways, that race and ethnicity indicators are not the only measures of diversity. Power dynamics, trust, and willingness to cooperate are often influenced by the “backstory,” which we must seek to understand.
Diversity is a worldview. How organizations and individuals understand and value diversity is important. Diversity is often considered in concert with culture and cultural practices. And culture is often described as either “surface” or “deep.” “Surface” cultural practices may include characteristics such as clothing, cuisine and art, while “deep” cultural practices may include behavior and value characteristics such as eye contact, speech patterns, concept of time, and notions of leadership and cooperation. As evaluators, working in culturally diverse settings requires the simultaneous consideration of both individuals’ beliefs and perspectives on “deep” cultural behaviors and values as well as organizations’ stances on diversity.
Communicating the interaction between individuals’ and organizations’ worldview in terms of diversity is a culturally responsive practice that informs why program improvement may or may not be evidenced.
We recommend the following resources as useful to amplifying our definitions of diversity: Faheemah Mustafaa’s post: Pursuing Racial Equity in Evaluation Practice.
Greetings, I am D. Pearl Barnett, MPA, graduate student at the University of Oklahoma, and recent graduate of AEA’s Graduate Education Diversity Internship (GEDI) program. During my work at the Urban League of Greater Oklahoma City, we completed an evaluation of the agency’s Strategic Plan. Here, I present a few salient lessons about cultural responsiveness from my experience.
When faced with the matter of “diversity” among evaluation stakeholders and participants, these questions surfaced: Is diversity an issue when organization staff reflect their community and service recipients (clients)? Does it matter in a monoracial organization/community? The theory of representative bureaucracy holds that shared demographics among stakeholders, staff, and clients, means shared values. By this logic, organizational programs, policies, and actvities should inherently address clients’ diverse needs and desires. The truth is organizations always have diversity concerns no matter its demographic make-up. It is our duty to address these concerns in our evaluations.
- Improve evaluation quality – and the relationship between organization and evaluation staff – by discussing the value of client input and the importance of allowing the data to speak for itself.
Representative bureaucracy assumes a link between demographic similarity and cultural awareness within an organization. Cultural responsiveness is the realization that though staff and clients may be demographically similar, there are still many differences (i.e., needs and values) that must be addressed among clients. Ensuring clients have a voice is the first step.
- Develop needs assessments, making sure to include the staff and community in every area of the process.
The cultural knowledge of the staff is important and ensuring all necessary information is gathered is equally important. The needs assessments serve as tools to obtain information directly from the clients. Staff and community member interviews throughout survey development are to improve respondents’ understanding, maximize the response rate, while generating staff buy-in to the evaluation and its outcomes.
Rad Resource: Effectively Managing Nonprofit Organizations. Edited By Richard L. Edwards and John A. Yankey. Published in 2006. Washington, DC: NASW Press. This book addresses diversity management and new approaches to program evaluation within the context of nonprofit organizations.
I’m Pat Campbell, president of Campbell-Kibler Associates, Inc. Under NSF funding, Eric Jolly, president of the Science Museum of Minnesota and I, with the help of a lot of friends, have been generating research-based tips, such as those below, to improve the accuracy of data collection, the quality of the analysis and the appropriateness of the data collected over diverse populations.
- Ask for demographic information ONLY at the end of measures. There may be exceptions in cases for people with disabilities who will need accommodations in order to complete the measures.
- Have participants define their own race/ethnicity and disability status rather than having the identification done by data collectors or project/program staff. If a standard set of categories for race/ethnicity and/or disability is used, also, in an open-ended question, ask participants to indicate their own race/ethnicity and disability status.
- Have members of the target population review affective and psychosocial measures for clarity. Ask them what concepts they think are being measured. If what is being measured is obvious and there are sex, race, or disability stereotypes associated with the concepts, consider using a less obvious measure if an equally valid measure is available.
- Be aware that there can be heterogeneity within subgroups. For example, while people who are visually impaired, hearing impaired, and learning disabled are all classified as having disabilities, the differences among them are very large and it might be appropriate to disaggregate by different categories of disability.
- When race/ethnicity, gender, or disability status is used as an independent variable, specify the reason for its use and include the reason in documentation of the results.
- All populations are diverse: The diversity may be in terms of race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status…. It may be visible or invisible. Most likely in every group there isa multiplicity of diversities. High quality evaluations need to pay attention to the diversity of all populations being served.
- Each individual is diverse. As individuals, we have many demographic characteristics including our race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status…. Rather than focusing on only one demographic category, high quality evaluations need to determine which categories are integral to the evaluation and focus on them.
- Universal Design for Evaluation Checklist, 4th Edition. The title, says it all. Jennifer Sullivan-Sulewski, & June Gothberg have developed a short planning tool that helps evaluators include people of all ages and all abilities in evaluations.
- As soon as it goes live, we hope our website, Beyond Rigor will be another rad resource. Let me know (Campbell@campbell-kibler.com) if you would like to be notified when that happens.
The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.
1 Comment · Posted by jgothberg in Disabilities and Other Vulnerable Populations, Indigenous Peoples in Evaluation, International and Cross-cultural Evaluation, Organizational Learning and Evaluation Capacity Building
I’m Karen Anderson, AEA’s Diversity Coordinator Intern, and in this role I support AEA’s diversity programs, TIGs, and the AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group.
The baton has been passed from the Cultural Competence in Evaluation Task Force, the Statement developers, to the Cultural Competence in Evaluation Dissemination Working Group to translate the Statement from paper to practice. One strategy for its broader dissemination and use is integrating the Statement into the policies and procedures of organizations that conduct and commission evaluations.
The AEA Public Statement on Cultural Competence in Evaluation has several core concepts, including the implications culture has for all phases of evaluation, including staffing of evaluation efforts and ensuring that members of the evaluation team collectively demonstrate cultural competence in the context for each evaluation. How does an evaluation practitioner or commissioner begin to do this?
- The Organizational Cultural and Linguistic Competency Assessment Tool can be used to assess where organizations and individuals fall along the cultural competence spectrum, and to serve as a guide to identify training needs and areas to improve upon.
- Share the Statement and supplemental resources like Building Culturally Competent Organizations, Key Components to a Culturally Competent System, and It All Starts At The Front Desk with human resources and decision makers in organizations. Recommend the development of a cultural competence committee to monitor and make recommendations for policy revision, program development, and evaluation.
- Include cultural competence language in the development and response to requests for proposals (RFPs). Check out this post How to Spot a Lip Service Approach to Culturally Responsive Evaluation from Patricia Rogers and Jane Davison’s Genuine Evaluation blog for tips on pointing out when a client may not be walking the walk in relation to culture and program development, theory, and evaluation.
- If you or other employees at your organization belong to an AEA affiliate, organize an event at your office around the theory or practical applications of the Statement. The Atlanta Area AEA affiliate group hosted one recently, Taking a Stance Toward Culture: Cultural Competence in Evaluation. Reflections from the event can be found in the AEA Newsletter diversity article.
- Set up a series of lunch and learns to begin having dialogue with colleague to increase awareness and to encourage relationship building, or start a book club discussion using the Statement, and branch out to other reading material to light the spark for cultural competence in evaluation.
The American Evaluation Association will be celebrating Cultural Competence Week. The contributions all this week come from the Cultural Competence committee. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.
Hi, I am Robin Kelly; I work as an internal evaluator for the National Minority AIDS Council, which is a federally funded nongovernmental organization that provides capacity building assistance to community based organizations and health departments that have programs or interventions to address the HIV/AIDS epidemic. I mention it because we work in communities of color. In doing so, attention to culture, be it individual, or organizational, must be given paramount attention.
Also, I live and work in Washington, DC. , a city of hot summers, temperate falls, unpredictable winters, warm springs and varying political winds all year long. It is also an extraordinarily diverse city. From the people to the types of nongovernmental organizations that exist here, there are a plethora of cultures.
The phrase cultural intelligence (CQ) is used as we systematically address the evaluation of personal interests and interactions. The abbreviated meaning of the term is the capability to function effectively across various cultural contexts (national, ethnic, organizational, generational, etc.) with heightened awareness of the characteristics of those in that setting, be they organizational culture or individual. Some have referred to cultural intelligence as the sister to emotional intelligence.
Hot Tip: When you are placed in an international or local setting or working with foreign nationals, remember to flex your CQ skills. When working with individuals or groups who represent diverse cultures, remember that each participant has four distinct capabilities that you will need to be sensitive to:
- Drive – motivation
- Cognition- understanding
- Strategic outlook-awareness
- Action-behavior (in new settings)
Rad Resource: Consider using a tool to optimize your CQ. In addition to the tips above a tool will help to reframe or adjust to those with whom you wish to consult or interact or provide a service. To see a tool that I recommend, see SAMPLE CQ SELF-TEST. The complete assessment can be found in Building Cultural Intelligence by Richard D. Bucher.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
I am Frances Carter, a Public Policy PhD Candidate at the University of Maryland Baltimore County. As a member of AEA’s 2009-2010 cohort of GEDIs (pronounced JEDI; acronym for Graduate Education and Diversity Internship), I developed an evaluation plan for the Annie E. Casey Foundation’s Race Matter’s Toolkit. The Race Matters Toolkit (RMT) was designed to address and produce racially equitable results and opportunities for all children, families, and communities. The toolkit is based on assumptions that race matters in creating opportunities for all and that embedded racial inequities present the greatest barrier to equitable opportunities and results. By using the RMT, the Annie E. Casey Foundation’s goal is to help organizations make the case, shape the message, and do their work from the perspective that race matters. After being used for several years, there was a need to evaluate the RMT and its work.
Top 5 Lessons Learned in Developing the RMT Evaluation Plan:
- Evaluations in large non-profit and government organizations are often conducted by external evaluators and managed by evaluation teams within the organization.
- Research methodologies learned in academic environments need to take a more practical approach when applied to evaluations in results driven environments.
- Various evaluation styles (i.e. culturally relevant and responsive evaluation and participatory evaluation) were used to develop a comprehensive RMT evaluation proposal.
- Standard evaluation tools such as logic models, theories of change, and evaluation proposals are critical for new evaluators both to understand the program being evaluated and to develop appropriate evaluation planning.
- Work in academic, non-profit, government, and private organizations is often, and should continue to be, integrated to address social problems.
Hot Tip: When I meet current or future graduate students interested in evaluation, I point them to AEA’s programs for developing new and diverse evaluators. These include the GEDI program, AEA’s Type I and II Travel awards, and the various Topical Interest Groups within AEA, specifically the Multiethnic Issues in Evaluation TIG. After participating in the Pipeline Program in 2007 (which serves students local to the conference locale and is being revamped for 2011), presenting at Evaluation 2008 and 2009 conferences, and being a GEDI, I gained valuable evaluation experience. I feel completely aware of and engaged in the AEA community as well as national and local evaluation communities with understanding for how evaluation supports addressing important social problems in the United States and internationally.
Rad Resource: The Annie E. Casey Foundation has made the Race Matters Toolkit available for free online download. Please notify the Foundation at firstname.lastname@example.org if and when you use the toolkit to assist with the RMT evaluation efforts.
Rad Resource: RMT component and evaluation information are available in AEA’s eLibrary.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello! My name is Psyche Williams-Forson and I am an Associate Professor of American Studies at the University of Maryland College Park. Working under the themes of the cultures of everyday life and constructions of identity and difference, I study the material world through gender, food behaviors, and power. When asked some of the ways evaluators can grapple with the thorny terrains of power, I respond with lessons learned from a dinner party.
Several years ago, when I worked in Student Affairs, I participated in assessing our division’s plan for the inclusion and implementation of diversity vis-à-vis the university’s strategic plan. As one of three women of color on the committee I often left our meetings feeling marginalized. Shortly after the final report was submitted, I received an invitation “to celebrate” our committee’s work at the home of the Dean. As a junior administrator, I felt my attendance was more than requested—it was required. I responded affirmatively and was assured that I need not bring anything. When I arrived, I was greeted with the main dish—tofu kabobs! One of the other women of color took me aside and asked, “What is this?” My response was, in a word—“power.”
Lessons Learned: A natural evaluation of this scenario reveals that in both a professional and personal setting, power was enacted. On the committee, members of the group left meetings believing that no institutionalized processes of change were forthcoming. While a lot of rhetoric around diversity was exchanged few concrete examples of institutional accountability emerged. Simply put, “the diversity box was checked”: meetings were held, and people of color were in attendance. Short of this, little systemic action was proposed.
Lessons Learned: Never dismiss the subtleties of power and how they are exercised, especially in relation to privilege. Foods, like the celebratory tofu, mark time, place, and circumstances by their absence as well as their presence. The host of the dinner party perhaps thought she was exercising proper etiquette when she suggested that we not bring anything to dinner. Yet, the “elephant in the room” was the obvious discomfort most of us felt being forced into a situation where we were to eat bean curd.
Hot tip: Power and privilege work in silence(s). Be attentive to the ways in which evaluation can help to reconstruct or reshape power relations.
Rad quote: “Most of the choices we make are unconscious…We can participate in systems in ways we’re not aware of and help produce consequences without knowing it…both historically and in the present, without any intention to do so.” (Johnson, “Who Me?” http://www.agjohnson.us/essays/whome )
Rad Resource: Privilege, Power, and Difference, 2e, by Allan G. Johnson (McGraw-Hill, 2005)