TAG | diversity
Hello and Hola. We’re Nnenia Campbell, research assistant at the Natural Hazards Center, University of Colorado Boulder and Saúl Maldonado, research assistant at the ELLISA Project, University of California, Santa Cruz. We are recent graduates of the AEA Graduate Diversity Education Internship (GEDI). One of the GEDI goals is to: “Deepen the evaluation profession’s capacity to work in racially, ethnically and culturally diverse settings.” In the first of this two-part series, we shared a few suggestions for amplifying definitions of diversity. In this second-part, we share a few of the lessons we’ve learned that frame diversity as a measure of validity in culturally responsive evaluation.
Diversity informs validity. Cultural responsiveness is a critical feature of the evaluation process and increases the utility and truthfulness of results (Frierson, Hood & Hughes, 2002). Practicing cultural responsiveness in settings that are racially, ethnically, culturally or otherwise diverse requires considering cultural contexts as technical properties of validity. As evaluators, positioning cultural contexts as validity measures may include review procedures such as the accuracy with which programs serve diverse populations as well as how evaluators select instrumentation and how participants of diverse cultural backgrounds interpret items.
Additionally, cultural responsiveness is co-constructed between evaluators and evaluands. We must seek to identify and communicate the diverse worldviews that are represented in the evaluation process and, more importantly, how diverse beliefs, perspectives, behaviors and values can serve as “resource differences.” Such communications should be continuous, throughout initial, intermediate and final stages of evaluations.
Instrument Selection. When deciding upon which tools and instruments will be used as evaluative measures, it is important to recognize the diversity of our professional training. As much as possible, a range of recommended tools and instruments from multi-disciplinary perspectives should be shared with program participants and organizations.
Item Interpretation. When evaluative measures deemed appropriate are co-selected by evaluators and evaluands, determining the degree of item’s construct-relevance is necessary. Practicing cultural responsiveness requires being attentive to and appreciative of the possibilities for cultural variation. Including participants’ life contexts, prevailing socio-economic conditions, cultural communication and socialization styles in relation to how items are interpreted may reveal the need to, for example, modify measures’ number of items or adapt measurement processes.
Review Procedures. After selecting appropriate instruments and establishing items’ construct-relevance, inclusionary procedures to establish representation of program participants’ and organizations’ diversity is needed. Integrating the perspectives from culturally diverse panels in the interpretation of findings and presentation of results enhances the accuracy and validity of an evaluation.
Regardless of evaluation approach (i.e., participatory, utilization, systems) or method, being attentive and appreciate of diverse cultural contexts enhances both internal and external validity. To reframe diversity as a measure of validity, we recommend the following resources: Frierson, Hood and Hughes’ A guide to conducting culturally responsive evaluation as well as Basterra, Trumbull and Solano-Flores’ book, Cultural validity in assessment: Addressing linguistic and cultural diversity.
Hello and Hola. We’re Nnenia Campbell, research assistant at the Natural Hazards Center, University of Colorado Boulder and Saúl Maldonado, research assistant at the ELLISA Project, University of California, Santa Cruz. We are recent graduates of the AEA Graduate Diversity Education Internship (GEDI). One of the GEDI goals is to: “Deepen the evaluation profession’s capacity to work in racially, ethnically and culturally diverse settings.” In this first of a two-part series, we share a few of the lessons we’ve learned that have led to our belief that definitions of diversity require amplification.
Diversity is not just race and ethnicity. Addressing the evaluative needs and interests of programs and organizations requires a consideration of the complex relationship between demographic composition and institutional factors. Program participants and organizations are never homogeneous. For example, factors such as age, gender, sexuality, language, religion, spirituality and national heritage as well as socio-economic and geospatial characteristics are significant traits that also intersect with racial and ethnic factors.
Additionally, it is important for evaluators to consider the relevance of political and/or socio-historical factors that may influence relationships between stakeholders. Paying attention to context offers insights to important underlying dynamics that influence the concerns expressed by key players. As evaluators, it is important to share with our stakeholders, in culturally responsive ways, that race and ethnicity indicators are not the only measures of diversity. Power dynamics, trust, and willingness to cooperate are often influenced by the “backstory,” which we must seek to understand.
Diversity is a worldview. How organizations and individuals understand and value diversity is important. Diversity is often considered in concert with culture and cultural practices. And culture is often described as either “surface” or “deep.” “Surface” cultural practices may include characteristics such as clothing, cuisine and art, while “deep” cultural practices may include behavior and value characteristics such as eye contact, speech patterns, concept of time, and notions of leadership and cooperation. As evaluators, working in culturally diverse settings requires the simultaneous consideration of both individuals’ beliefs and perspectives on “deep” cultural behaviors and values as well as organizations’ stances on diversity.
Communicating the interaction between individuals’ and organizations’ worldview in terms of diversity is a culturally responsive practice that informs why program improvement may or may not be evidenced.
We recommend the following resources as useful to amplifying our definitions of diversity: Faheemah Mustafaa’s post: Pursuing Racial Equity in Evaluation Practice.
Greetings, I am D. Pearl Barnett, MPA, graduate student at the University of Oklahoma, and recent graduate of AEA’s Graduate Education Diversity Internship (GEDI) program. During my work at the Urban League of Greater Oklahoma City, we completed an evaluation of the agency’s Strategic Plan. Here, I present a few salient lessons about cultural responsiveness from my experience.
When faced with the matter of “diversity” among evaluation stakeholders and participants, these questions surfaced: Is diversity an issue when organization staff reflect their community and service recipients (clients)? Does it matter in a monoracial organization/community? The theory of representative bureaucracy holds that shared demographics among stakeholders, staff, and clients, means shared values. By this logic, organizational programs, policies, and actvities should inherently address clients’ diverse needs and desires. The truth is organizations always have diversity concerns no matter its demographic make-up. It is our duty to address these concerns in our evaluations.
- Improve evaluation quality – and the relationship between organization and evaluation staff – by discussing the value of client input and the importance of allowing the data to speak for itself.
Representative bureaucracy assumes a link between demographic similarity and cultural awareness within an organization. Cultural responsiveness is the realization that though staff and clients may be demographically similar, there are still many differences (i.e., needs and values) that must be addressed among clients. Ensuring clients have a voice is the first step.
- Develop needs assessments, making sure to include the staff and community in every area of the process.
The cultural knowledge of the staff is important and ensuring all necessary information is gathered is equally important. The needs assessments serve as tools to obtain information directly from the clients. Staff and community member interviews throughout survey development are to improve respondents’ understanding, maximize the response rate, while generating staff buy-in to the evaluation and its outcomes.
Rad Resource: Effectively Managing Nonprofit Organizations. Edited By Richard L. Edwards and John A. Yankey. Published in 2006. Washington, DC: NASW Press. This book addresses diversity management and new approaches to program evaluation within the context of nonprofit organizations.
I’m Pat Campbell, president of Campbell-Kibler Associates, Inc. Under NSF funding, Eric Jolly, president of the Science Museum of Minnesota and I, with the help of a lot of friends, have been generating research-based tips, such as those below, to improve the accuracy of data collection, the quality of the analysis and the appropriateness of the data collected over diverse populations.
- Ask for demographic information ONLY at the end of measures. There may be exceptions in cases for people with disabilities who will need accommodations in order to complete the measures.
- Have participants define their own race/ethnicity and disability status rather than having the identification done by data collectors or project/program staff. If a standard set of categories for race/ethnicity and/or disability is used, also, in an open-ended question, ask participants to indicate their own race/ethnicity and disability status.
- Have members of the target population review affective and psychosocial measures for clarity. Ask them what concepts they think are being measured. If what is being measured is obvious and there are sex, race, or disability stereotypes associated with the concepts, consider using a less obvious measure if an equally valid measure is available.
- Be aware that there can be heterogeneity within subgroups. For example, while people who are visually impaired, hearing impaired, and learning disabled are all classified as having disabilities, the differences among them are very large and it might be appropriate to disaggregate by different categories of disability.
- When race/ethnicity, gender, or disability status is used as an independent variable, specify the reason for its use and include the reason in documentation of the results.
- All populations are diverse: The diversity may be in terms of race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status…. It may be visible or invisible. Most likely in every group there isa multiplicity of diversities. High quality evaluations need to pay attention to the diversity of all populations being served.
- Each individual is diverse. As individuals, we have many demographic characteristics including our race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status…. Rather than focusing on only one demographic category, high quality evaluations need to determine which categories are integral to the evaluation and focus on them.
- Universal Design for Evaluation Checklist, 4th Edition. The title, says it all. Jennifer Sullivan-Sulewski, & June Gothberg have developed a short planning tool that helps evaluators include people of all ages and all abilities in evaluations.
- As soon as it goes live, we hope our website, Beyond Rigor will be another rad resource. Let me know (Campbell@campbell-kibler.com) if you would like to be notified when that happens.
The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.
1 Comment · Posted by jgothberg in Disabilities and Other Vulnerable Populations, Indigenous Peoples in Evaluation, International and Cross-cultural Evaluation, Organizational Learning and Evaluation Capacity Building
I’m Karen Anderson, AEA’s Diversity Coordinator Intern, and in this role I support AEA’s diversity programs, TIGs, and the AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group.
The baton has been passed from the Cultural Competence in Evaluation Task Force, the Statement developers, to the Cultural Competence in Evaluation Dissemination Working Group to translate the Statement from paper to practice. One strategy for its broader dissemination and use is integrating the Statement into the policies and procedures of organizations that conduct and commission evaluations.
The AEA Public Statement on Cultural Competence in Evaluation has several core concepts, including the implications culture has for all phases of evaluation, including staffing of evaluation efforts and ensuring that members of the evaluation team collectively demonstrate cultural competence in the context for each evaluation. How does an evaluation practitioner or commissioner begin to do this?
- The Organizational Cultural and Linguistic Competency Assessment Tool can be used to assess where organizations and individuals fall along the cultural competence spectrum, and to serve as a guide to identify training needs and areas to improve upon.
- Share the Statement and supplemental resources like Building Culturally Competent Organizations, Key Components to a Culturally Competent System, and It All Starts At The Front Desk with human resources and decision makers in organizations. Recommend the development of a cultural competence committee to monitor and make recommendations for policy revision, program development, and evaluation.
- Include cultural competence language in the development and response to requests for proposals (RFPs). Check out this post How to Spot a Lip Service Approach to Culturally Responsive Evaluation from Patricia Rogers and Jane Davison’s Genuine Evaluation blog for tips on pointing out when a client may not be walking the walk in relation to culture and program development, theory, and evaluation.
- If you or other employees at your organization belong to an AEA affiliate, organize an event at your office around the theory or practical applications of the Statement. The Atlanta Area AEA affiliate group hosted one recently, Taking a Stance Toward Culture: Cultural Competence in Evaluation. Reflections from the event can be found in the AEA Newsletter diversity article.
- Set up a series of lunch and learns to begin having dialogue with colleague to increase awareness and to encourage relationship building, or start a book club discussion using the Statement, and branch out to other reading material to light the spark for cultural competence in evaluation.
The American Evaluation Association will be celebrating Cultural Competence Week. The contributions all this week come from the Cultural Competence committee. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.
Hi, I am Robin Kelly; I work as an internal evaluator for the National Minority AIDS Council, which is a federally funded nongovernmental organization that provides capacity building assistance to community based organizations and health departments that have programs or interventions to address the HIV/AIDS epidemic. I mention it because we work in communities of color. In doing so, attention to culture, be it individual, or organizational, must be given paramount attention.
Also, I live and work in Washington, DC. , a city of hot summers, temperate falls, unpredictable winters, warm springs and varying political winds all year long. It is also an extraordinarily diverse city. From the people to the types of nongovernmental organizations that exist here, there are a plethora of cultures.
The phrase cultural intelligence (CQ) is used as we systematically address the evaluation of personal interests and interactions. The abbreviated meaning of the term is the capability to function effectively across various cultural contexts (national, ethnic, organizational, generational, etc.) with heightened awareness of the characteristics of those in that setting, be they organizational culture or individual. Some have referred to cultural intelligence as the sister to emotional intelligence.
Hot Tip: When you are placed in an international or local setting or working with foreign nationals, remember to flex your CQ skills. When working with individuals or groups who represent diverse cultures, remember that each participant has four distinct capabilities that you will need to be sensitive to:
- Drive – motivation
- Cognition- understanding
- Strategic outlook-awareness
- Action-behavior (in new settings)
Rad Resource: Consider using a tool to optimize your CQ. In addition to the tips above a tool will help to reframe or adjust to those with whom you wish to consult or interact or provide a service. To see a tool that I recommend, see SAMPLE CQ SELF-TEST. The complete assessment can be found in Building Cultural Intelligence by Richard D. Bucher.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
I am Frances Carter, a Public Policy PhD Candidate at the University of Maryland Baltimore County. As a member of AEA’s 2009-2010 cohort of GEDIs (pronounced JEDI; acronym for Graduate Education and Diversity Internship), I developed an evaluation plan for the Annie E. Casey Foundation’s Race Matter’s Toolkit. The Race Matters Toolkit (RMT) was designed to address and produce racially equitable results and opportunities for all children, families, and communities. The toolkit is based on assumptions that race matters in creating opportunities for all and that embedded racial inequities present the greatest barrier to equitable opportunities and results. By using the RMT, the Annie E. Casey Foundation’s goal is to help organizations make the case, shape the message, and do their work from the perspective that race matters. After being used for several years, there was a need to evaluate the RMT and its work.
Top 5 Lessons Learned in Developing the RMT Evaluation Plan:
- Evaluations in large non-profit and government organizations are often conducted by external evaluators and managed by evaluation teams within the organization.
- Research methodologies learned in academic environments need to take a more practical approach when applied to evaluations in results driven environments.
- Various evaluation styles (i.e. culturally relevant and responsive evaluation and participatory evaluation) were used to develop a comprehensive RMT evaluation proposal.
- Standard evaluation tools such as logic models, theories of change, and evaluation proposals are critical for new evaluators both to understand the program being evaluated and to develop appropriate evaluation planning.
- Work in academic, non-profit, government, and private organizations is often, and should continue to be, integrated to address social problems.
Hot Tip: When I meet current or future graduate students interested in evaluation, I point them to AEA’s programs for developing new and diverse evaluators. These include the GEDI program, AEA’s Type I and II Travel awards, and the various Topical Interest Groups within AEA, specifically the Multiethnic Issues in Evaluation TIG. After participating in the Pipeline Program in 2007 (which serves students local to the conference locale and is being revamped for 2011), presenting at Evaluation 2008 and 2009 conferences, and being a GEDI, I gained valuable evaluation experience. I feel completely aware of and engaged in the AEA community as well as national and local evaluation communities with understanding for how evaluation supports addressing important social problems in the United States and internationally.
Rad Resource: The Annie E. Casey Foundation has made the Race Matters Toolkit available for free online download. Please notify the Foundation at email@example.com if and when you use the toolkit to assist with the RMT evaluation efforts.
Rad Resource: RMT component and evaluation information are available in AEA’s eLibrary.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello! My name is Psyche Williams-Forson and I am an Associate Professor of American Studies at the University of Maryland College Park. Working under the themes of the cultures of everyday life and constructions of identity and difference, I study the material world through gender, food behaviors, and power. When asked some of the ways evaluators can grapple with the thorny terrains of power, I respond with lessons learned from a dinner party.
Several years ago, when I worked in Student Affairs, I participated in assessing our division’s plan for the inclusion and implementation of diversity vis-à-vis the university’s strategic plan. As one of three women of color on the committee I often left our meetings feeling marginalized. Shortly after the final report was submitted, I received an invitation “to celebrate” our committee’s work at the home of the Dean. As a junior administrator, I felt my attendance was more than requested—it was required. I responded affirmatively and was assured that I need not bring anything. When I arrived, I was greeted with the main dish—tofu kabobs! One of the other women of color took me aside and asked, “What is this?” My response was, in a word—“power.”
Lessons Learned: A natural evaluation of this scenario reveals that in both a professional and personal setting, power was enacted. On the committee, members of the group left meetings believing that no institutionalized processes of change were forthcoming. While a lot of rhetoric around diversity was exchanged few concrete examples of institutional accountability emerged. Simply put, “the diversity box was checked”: meetings were held, and people of color were in attendance. Short of this, little systemic action was proposed.
Lessons Learned: Never dismiss the subtleties of power and how they are exercised, especially in relation to privilege. Foods, like the celebratory tofu, mark time, place, and circumstances by their absence as well as their presence. The host of the dinner party perhaps thought she was exercising proper etiquette when she suggested that we not bring anything to dinner. Yet, the “elephant in the room” was the obvious discomfort most of us felt being forced into a situation where we were to eat bean curd.
Hot tip: Power and privilege work in silence(s). Be attentive to the ways in which evaluation can help to reconstruct or reshape power relations.
Rad quote: “Most of the choices we make are unconscious…We can participate in systems in ways we’re not aware of and help produce consequences without knowing it…both historically and in the present, without any intention to do so.” (Johnson, “Who Me?” http://www.agjohnson.us/essays/whome )
Rad Resource: Privilege, Power, and Difference, 2e, by Allan G. Johnson (McGraw-Hill, 2005)
1 Comment · Posted by Susan Kistler in Disabilities and Other Vulnerable Populations, Indigenous Peoples in Evaluation, Multiethnic Issues in Evaluation, Organizational Learning and Evaluation Capacity Building
My name is Gerri Spilka and I am the Executive Director of the OMG Center for Collaborative Learning – a consulting firm that assists and assesses change through evaluation, strategic research and planning, grants management and capacity building. Among our many projects, OMG serves as the lead management team on the Robert Wood Johnson Foundation Evaluation Fellowship Program and it is about that program that I will share information today.
The skill set needed to interpret effectively public health, social service, community and organizational social patterns depends on the authentic and diverse experiences within program evaluation teams. Programs and services that are often evaluated traditionally work with underserved and disadvantaged communities, which are themselves becoming more diverse. However, the program evaluation field is becoming less diverse due to a lack of training and placement opportunities for emerging professionals from graduate programs who come from underserved and disadvantaged communities. This creates a problem for the field where, more often than not, high-quality evaluations reflect a process that incorporates diverse perspectives. The RWJF Evaluation Fellowship Program works to ameliorate this disparity.
Resource: The RWJF Evaluation Fellowship Program seeks to extend the skills of participants from diverse backgrounds to include culturally responsive evaluation practices. The Fellows program supports two cohorts annually on parallel paths:
- The Emerging Professionals Program is aimed at those who have graduated from a master’s or doctoral degree program within the last three years and provides a one-year placement, extensive training opportunities, and a support stipend
- The Retooling Professionals Program is focused on mid- or senior-level nonprofit professionals who remain in their current position, but receive additional training to extend their skills to evaluation. Their organization receives a $5,000 financial award and the professional receives a travel stipend.
Both cohorts receive customized training, facilitated by some of today’s thought leaders in field, as well as mentorship, and practical field-based guidance.
Hot Tip: While the deadline from participating in the Retooling Professionals program has passed, the deadline for applying for the Emerging Professionals program is coming up on July 31, 2010. The Emerging Professionals application may be downloaded online from http://www.rwjf-evaluationfellows.org/emerging-fellows.
Resource: The RWJF Evaluation Fellowship Program just released a video entitled Evaluation With a Diversity Lens. This extended discourse among a range of professionals explores the value of diversity in the evaluation profession and may be downloaded for free.