AEA365 | A Tip-a-Day by and for Evaluators

TAG | culture

Greetings, I am June Gothberg, incoming Director of the Michigan Transition Outcomes Project and past co-chair of the Disabilities and Other Vulnerable Populations topical interest group at AEA.  I hope you’ve enjoyed a great week of information specific to projects involving these populations.  As a wrap up I thought I’d end with broad information on involving vulnerable populations in your evaluation and research projects.

Lessons Learned: Definition of “vulnerable population”

  • The TIGs big ah-ha.  When I came in as TIG co-chair, I conducted a content analysis of the presentations of our TIG for the past 25 years.  We had a big ah-ha when we realized what and who is identified as “vulnerable populations”.  The list included:
    • Abused
    • Abusers
    • Chronically ill
    • Culturally different
    • Economically disadvantaged
    • Educationally disadvantaged
    • Elderly
    • Foster care
    • Homeless
    • Illiterate
    • Indigenous
    • Mentally ill
    • Migrants
    • Minorities
    • People with disabilities
    • Prisoners
    • Second language
    • Veterans – “wounded warriors”
  • Determining vulnerability.  The University of South Florida provides the following to determine vulnerability in research:
    • Any individual that due to conditions, either acute or chronic, who has his/her ability to make fully informed decisions for him/herself diminished can be considered vulnerable.
    • Any population that due to circumstances, may be vulnerable to coercion or undue influence to participate in research projects.

vulnerable

Hot Tips:  Considerations for including vulnerable populations.

  • Procedures.  Use procedures to protect and honor participant rights.
  • Protection.  Use procedures to minimize the possibility of participant coercion or undue influence.
  • Accommodation.  Prior to start, make sure to determine and disseminate how participants will be accommodated in regards to recruitment, informed consent, protocols and questions asked, retention, and research procedures including those with literacy, communication, and second language needs.
  • Risk.  Minimize any unnecessary risk to participation.

Hot Tips:  When your study is targeted at vulnerable populations.

  • Use members of targeted group to recruit and retain subjects.
  • Collaborate with community programs and gatekeepers to share resources and information.
  • Know the formal and informal community.
  • Examine cultural beliefs, norms, and values.
  • Disseminate materials and results in an appropriate manner for the participant population.

Rad Resources:

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

Hi. I am Don Glass the co-chairperson of the Arts, Culture, and Audiences TIG. Along with my co-chair Kathleen Tinworth, we are excited to host a week of aea365. Our colleague Joe Heimlich has curated this series to explore evaluation of arts and culture, arts education, arts participation, informal learning, and visitor studies. Please join us this week using the comments feature of aea365 to think about how this work may have implications for your work in the field of evaluation!

Arts, culture, and arts education may have outcomes and outputs that are often creative, innovative, and unpredictable. These may have deep personal and cultural meanings and value to various stakeholders. To understand and evaluate the arts, evaluators tend to explore more naturalistic and qualitative forms of inquiry and reporting. These methods can be rigorous and systematic, as well as allow for responsiveness to the context of the cultural values and aesthetic qualities of a program.

Let’s start the week by visiting two foundational lessons that emerged from the evaluation of arts education, and then look at several resources that feature recent evaluation thinking and tools inspired by engagement with arts and culture.

Lessons Learned:

  • Expressive Outcomes: Elliot Eisner in his 1972 article Emerging Models for Educational Evaluation, speaks of expressive types of outcomes that are not predetermined, but are generated out of an activity and then reflected upon through a responsive evaluative lens.
  • Responsive Evaluation: In Robert Stake’s 1975 classic To Evaluate an Arts Program, he cautioned us about an “over-reliance on per-conceived notions of success, ” (p. 15) and recommends that we pay close attention, observe carefully, and portray the complexity in ways that are responsive to what people are naturally doing.

Rad Resources:

Evaluating Arts Education Programs:

Evaluating Arts Organizations/Programs:

Evaluating Arts Participation:

The American Evaluation Association is celebrating Arts, Culture, and Audiences (ACA) TIG Week. The contributions all week come from ACA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

I’m Kelly Hannum. I’ve been evaluating leadership development programs for almost two decades. I am convinced that effective leaders and effective evaluators have similar mindsets and employ similar skills.  I encourage leaders to think like evaluators, and via this post I’m encouraging evaluators to think and develop themselves as leaders.

At the Center for Creative Leadership, we say “effective leadership results in shared direction, alignment, and commitment”. Leaders help focus people on defining and achieving something of shared value, but effective leadership is often a collective act.  How often you have worked with diverse stakeholders to create shared direction, alignment, and commitment related to an evaluation? Stakeholders often have different values and perspectives. Our role as evaluators is to effectively and respectfully lead these complex situations in a manner that reflects our Guiding Principles. What does “value” look like from different perspectives? What types of evidence of “value” are appropriate? Our training and experience is a powerful asset, but if left unchecked our assumptions can be a liability. Thinking of, and developing, ourselves as leaders can help us improve our evaluation practice.

Lessons Learned:

Be curious about yourself. Self-awareness is the foundation for being a good leader and for being a good evaluator. Understand your assets and limitations, plan accordingly, and continue to develop yourself. Challenge assumptions that may get in the way of understanding value from different perspectives.  Seek, consider, and apply feedback about yourself.

Be curious about others. Pay attention to other perspectives, that is the foundation for respect and understanding of complex situations. Examine and reexamine perceptions and beliefs, assumptions or stereotypes, about individuals, groups, and even programs and processes. Seek different perspectives and listen with curiosity and openness.

Hot Tip:

Reflect on how you create shared direction, alignment, and commitment. Think about keeping a journal or having informal conversations or debriefs after key meetings.

Most successful development experiences contain elements of assessment, challenge, and support – are you balanced? What do you need to add or reduce?

  • Assess yourself from different perspectives to uncover areas of excellence as well as areas for growth
  • Challenge yourself by learning about and trying new things
  • Get the support you need to be effective

Rad Resources:

Track your reflections using a free online journal like Penzu.

The Leadership Learning Community offers a collection of free leadership development resources including evaluation of leadership development.

The Center for Creative Leadership offers free articles and podcasts. The white papers are particularly helpful.

AEA’s Statement on Cultural Competence in Evaluation provides an overview of cultural competence, why it is important, and how to develop it.

Flipping the Script: White Privilege and Community Building is useful to my work.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hi, I am Robin Kelly; I work as an internal evaluator for the National Minority AIDS Council, which is a federally funded nongovernmental organization that provides capacity building assistance to community based organizations and health departments that have programs or interventions to address the HIV/AIDS epidemic. I mention it because we work in communities of color. In doing so, attention to culture, be it individual, or organizational, must be given paramount attention.

Also, I live and work in Washington, DC. , a  city of hot summers, temperate falls, unpredictable winters, warm springs and varying political winds all year long. It is also an extraordinarily diverse city.  From the people to the types of nongovernmental organizations that exist here, there are a plethora of cultures.

The phrase cultural intelligence (CQ) is used as we systematically address the evaluation of personal interests and interactions. The abbreviated meaning of the term is the capability to function effectively across various cultural contexts (national, ethnic, organizational, generational, etc.) with heightened awareness of the characteristics of those in that setting, be they organizational culture or individual.  Some have referred to cultural intelligence as the sister to emotional intelligence.

Hot Tip: When you are placed in an international or local setting or working with foreign nationals, remember to flex your CQ skills.  When working with individuals or groups who represent diverse cultures, remember that each participant has four distinct capabilities that you will need to be sensitive to:

  1. Drive – motivation
  2. Cognition- understanding
  3. Strategic outlook-awareness
  4.  Action-behavior (in new settings)

Rad Resource: Consider using a tool to optimize your CQ. In addition to the tips above a tool will help to reframe or adjust to those with whom you wish to consult or interact or provide a service.  To see a tool that I recommend, see SAMPLE CQ SELF-TEST.  The complete assessment can be found in Building Cultural Intelligence by Richard D. Bucher.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

I’m Dominica McBride, President of The HELP Institute, Inc. Recently, I wrote an article for New Directions for Evaluation on Sociocultural Theory (ST) and its applicability to evaluation.

Joke: There were a few young fish swimming near an older, wise fish and the old, wise fish says, “My, isn’t the water wonderful today?” and the young fish say, “Water? What water?”

Relevance: Culture is like water to fish – influencing our lives but often taken for granted or never even seen. It can be this ethereal and abstract concept. However, reflecting on and examining culture can be pivotal in personal and professional growth, program improvement, and societal change.

ST is one of the most comprehensive models on culture and human development, touching on all aspects of culture – the biological, psychological, interpersonal, linguistic, ecological, and historical. It provides simple guiding principles for evaluation and research practice:

Hot Tip: The unit of analysis is the sociocultural activity. Examine naturally occurring activities as it relates to the program or process (e.g., participants interacting). These activities should be considered in the context of the program, sociopolitical environment, family norms, organizational culture, etc.

Hot Tip: To understand a person, group, or social phenomena, we must ascertain the ever-changing environment and acknowledge and examine the development of the person, group, or program over time.

Hot Tip: Individual dynamics are affected by intrapersonal, interpersonal, and community dynamics – all of which are inseparable. So, to study a person or program, we must also consider the multifaceted influences that affect the participant behavior and the program.

Hot Tip: Groups have more variety within than between, which means there is more “cultural group” within a group and we must take this into consideration when learning of “another culture.” This fact can also help to dispel stereotypes. There are also many commonalities between groups. These phenomena help us to see and appreciate both the differences and links between us and others.

Hot Tip: Often times, we can get attached to methods. ST reminds us that the question should drive the methods and not the other way around. This assumption also encompasses the need for interdisciplinary work, opening our minds and hearts to other professionals and ways of doing things.

Hot Tip: In order to truly understand another, we must understand ourselves. Thus, we should take time to reflect on our development, cultural influences, personal and professional context, and intrapersonal dynamics. Without cleaning our own lens, we will always see others through tainted glasses.

Rad Resource:

Sociocultural Theory expert Barara Rogoff’s comprehensive book, The Cultural Nature of Human Development.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Kelly Robertson and I am a project manager at The Evaluation Center and a student in the Interdisciplinary Ph.D. in Evaluation program at Western Michigan University. This fall I was fortunate enough to have been able to attend several lectures and discussions led by visiting scholar Dr. Rodney Hopson which left me with two thought provoking questions regarding the depths of culture and what it looks like in practice.

Lesson Learned: Where’s culture if the standard –isms are not of primary focus? AEA’s Public Statement on Cultural Competence in Evaluation defines culture as, “the shared experiences of people, including their languages, values, customs, beliefs, and mores. It also includes worldviews, ways of knowing, and ways of communicating.” Most often when I think of culture I think of the examples cited in the statement, “race/ethnicity, religion, social class, language, disability, sexual orientation, age, and gender.” When Dr. Hopson asked staff to briefly describe the role of culture within each of our projects, I found it difficult to identify culture in projects that did not explicitly deal with the most commonly referenced –isms. Later different aspects of culture started to become more salient such as the culture of organizations I work with in which evaluations and methods are mandated and how my communication with clients sometimes differs based on the culture of their respective organizations. I also began to think about what it means, in some cases, if certain questions about the common –isms are not addressed within the work of evaluands, do unasked questions say something about culture? This thought process got me to realize that culture is a part of everything we do and that I need to engage in more self-reflection about my own cultures to thoroughly become aware of the cultures around me.

Lesson Learned: How can engagement in a culturally competent process be demonstrated in evaluation reports and incorporated into metaevaluation? After listening to Drs. Hopson and Stufflebeam discuss the Program Evaluation Standards, I began to wonder how evaluators can document that they have engaged in a culturally competent process and how to determine if other evaluators did the same from written reports. What are the criteria and indicators that should be used to make such a determination? Further, is it possible for an evaluator to demonstrate both independence while at the same time engaging in a participatory evaluation, at a level beyond just participation in data collection, as is often suggested in reference to a culturally competent process?

Rad Resources:

Here’s the public statement on cultural competence: http://www.eval.org/ccstatement.asp

While we should all own a copy of the Program Evaluation Standards, they have been summarized online and can be freely accessed here: http://www.eval.org/evaluationdocuments/progeval.html

All this week, we’re highlighting posts from colleagues at Western Michigan University as they reflect on a recent visit from incoming AEA President Rodney Hopson. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello, I’m Linda Cabral, a Senior Project Director from the Center for Health Policy and Research at the University of Massachusetts Medical School.

In an effort to more fully engage people from different cultural backgrounds and their communities in one of our recently-completed qualitative evaluation projects for our State’s Department of Mental Health, we employed the use of cultural brokers as members of our evaluation team. Cultural brokering has been defined as the act of bridging, linking or mediating between groups or persons of differing cultural backgrounds for the purpose of reducing conflict or producing change (National Center for Cultural Competence, 2004). In our case, we were seeking information from people with mental health conditions from specific population groups: Latinos and persons who are deaf or hard of hearing. We brought to our evaluation team cultural brokers who were people with mental health conditions and who were also members of the cultural groups we were interested in. They helped us develop our data collection instruments, led recruitment efforts, and participated in the data collection and data analysis phases.

Lesson Learned: The cultural brokers were able to establish a rapport and level of trust with study participants that would have been impossible to otherwise achieve. This rapport was important not only during the recruitment phase, but also during the data collection itself, thereby improving the quality of the data collected.

Lesson Learned: A barrier often cited with collecting data from non-English speakers is the need for interpreters. By using cultural brokers, participants were able to communicate as they felt most comfortable. Consider the use of cultural brokers when exploring sensitive topics with people from different cultural groups.

Lesson Learned: As the cultural brokers had little to no experience with evaluation work, it was necessary to build in time to educate the cultural brokers on evaluation basics. This helped to make our cultural brokers feel like a fully participating team member.

Rad Resource: The National Center for Cultural Competence (http://nccc.georgetown.edu/) has a host of resources to help programs design, implement, and evaluate culturally- and linguistically-competent service delivery systems.

Rad Resource: For those of you interested in using cultural brokers in the mental health field, the following article might be useful.

Singh NN, McKay JD, and Singh AN. (1999) The need for cultural brokers in mental health services. Journal of Child and Family Studies, 8(1):1-10.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Greetings from Champaign, IL! We are Ayesha Boyce, Maria Jimenez, and Gabriela Juarez. We are all currently pursuing our Ph.D. in evaluation at the University of Illinois Urbana-Champaign. We have found as a multicultural evaluation team that sometimes sensitive topics such as race, class and gender are best presented through creative means. These types of presentations can be used in conjunction with formal evaluation reporting.

Lesson Learned: Some of these creative presentations can include skits, poems, music and dance. We learned about and how to facilitate discussion with stakeholders on sensitive topics using the above methods in a number of Dr. Jennifer Greene’s graduate courses. After the call for 2011 Presidential Stand Proposals we wanted to contribute our knowledge and experiences in using the alternative method of skit.

Hot Tip: Know the evaluation context. Because culture can affect program design, implementation and outcomes, all evaluators must take into account the cultural and contextual characteristics of an evaluation. Some techniques to gain a better understanding of the context include: observing the program environment, engaging program staff in discussion about program design, and informally interviewing multiple stakeholders (i.e. participants, funders, staff and community members).

Hot Tip: Know participant’s culture.  Attending to the culture of a program and its participants, provides a more comprehensive understanding of program outcomes, which ultimately makes for a more useful evaluation for all stakeholders. Some techniques to gain a better understanding of a participant’s culture include: creating a diverse evaluation team where at least one person is familiar with the culture of participants and having informal conversations with participants and staff about culture and other sensitive topics.

Hot Tip: Know how to present sensitive information. Conveying participant values to stakeholders who may have their own set of differing values can be daunting. Evaluators can present participant values, particularly sensitive topics, in a creative way that fully showcases these complex dimensions. This creative and less formal presentation can diffuse tensions often associated with competing values and cultural misconceptions.

Hot Tip: Know how to write a skit. When presenting evaluation findings in a skit format:1) Begin the skit with a general setting that is similar to the evaluation’s context;  2) Create a dialogue within the skit around the pertinent evaluation findings; 3) Utilize stakeholders in the presentation (i.e. actors in the skit); and 4) Have a discussion at the end of the skit.

Rad References:

Frierson, H.T., Hood, S., & Hughes, G.B. (2002). Strategies that address culturally responsive evaluation. In the 2002 user-friendly handbook for project evaluation, 63-73. Arlington, Virginia, National Science Foundation.

Greene, J.C., Boyce, A.S, & Ahn, J. (2011). Value-Engaged, Educative Evaluation Guidebook. University of Illinois, Urbana-Champaign. Created and produced withfunds from the National Science Foundation.

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from this team? They’ll be presenting as part of the Evaluation 2011 Conference Program, November 2-5 in Anaheim, California.

· · ·

This post is brought to you by Alyssa Na’im of the National Science Foundation’s ITEST Learning Resource Center at the Education Development Center, and several members of the ITEST Community of Practice: Angelique Tucker Blackmon, Innovative Learning Concepts, LLC; Araceli M. Ortiz, Sustainable Future, Inc.; Carol Nixon, Edvantia, Inc.; Pam Van Dyk, Evaluation Resources, LLC; and Karen L. Yanowitz, Arkansas State University. ITEST stands for Innovative Technology Experience for Students and Teachers, and was established in 2003 to address concerns about the growing demand for science, technology, engineering, and mathematics (STEM) professionals in the U.S.  The ITEST program helps young people and teachers in formal and informal K-12 settings build the knowledge and skills needed to succeed in a technologically rich society.

Rad Resources: We hosted a session (slides) at Evaluation 2010 and a webinar for the ITEST community (slides) that explored issues relating to culture, context, and stakeholder engagement in evaluation and wanted to share these insights with the AEA365 community.

Lesson Learned: Evaluators’ understanding of stakeholders’ cultural contexts should frame the way they engage and communicate with stakeholders as well as inform their professional practice.

The definition of “culture” includes not only typical reference to beliefs, social norms, and practices of racial, ethnic, religious, and/or social groups, but also references to values, goals, and practices of an institution or organization as well as those of a particular field or discipline. This provides a macro-level definition, reflecting all stakeholders involved in a program and its evaluation. Stakeholders are those who are invested in the program and are affected by its outcomes. We identify stakeholders as belonging to one or more of three groups: decision makers (e.g., funders, principal, director), implementers (e.g., staff, teachers), and recipients (e.g., students, parents, community).

Responding to stakeholders and involving them in the evaluation requires the evaluator to balance multiple goals.

  • Stakeholders may have different evaluation needs. Funders typically are more interested in summative results showing impact of the program, while implementers are often additionally concerned with formative information to guide program development.
  • Evaluation design and data collection methods should accommodate not just language, age, and developmental requirements, but also situational contexts.
  • Communication and reporting depends on the needs of the stakeholder. While all stakeholders should receive some information regarding program outcomes, what they receive, when they receive it, and the degree of detail that they receive depends on the goals of the program.

Acknowledging that each stakeholder’s perspective emerges from their culture and context, and striving to better understand their perspectives enhances evaluators’ abilities to relate to and engage multiple stakeholders in the evaluation process. Evaluators must be active, reactive, and adaptive participants in the evaluation to effectively engage all stakeholders.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

My name is Hanife Cakici and I am a second year graduate student in the Master of Public Policy program at the Humphrey School of Public Affairs, University of Minnesota, focusing on advanced policy analysis and program evaluation. Coming to U.S as a Turkish Fulbright student has generated a deep curiosity in me to investigate how to foster evaluation capacity building and organizational development in the developing world.

Carol Weiss’s four I’s “Ideology, Interests, Information, and Institution” nicely capture the intricate web of relations and domains through which the evaluation evolves. These four I’s pose even more challenge for evaluations conducted in culturally, ethnically, and linguistically distinct environments. In order to sustain political credibility in the midst of multiple threats to independence, the evaluator must have technical skills. But where to start?

Having lived in several countries over the last 5 years, I have confirmed that the books listed below will provide evaluators with sufficient information about the dynamics shaping both the developed and developing world. You may find the authors’ arguments controversial or provocative. Yet they will provide evaluators with enough material to contemplate when they need to evaluate a program that is a unique product of history, religion, culture, and geography.

Hot Tip- Hot ‘Book’ 1: The Clash of Civilizations by Samuel Huntington tackles the cultural differences, and potential for conflicts between groups of civilizations based on historical, political, social and religious characteristics. The book will provide evaluators with a systems approach, and also help acknowledge and appreciate the ‘similarities’ between cultures rather than the differences while they are designing evaluations for distinct populations around the world.

Hot Tip- Hot ‘Book’ 2: Three Worlds of Welfare Capitalism by Esping-Anderson presents three types of welfare states: Liberal Model (e.g. U.S.A, Canada, and Australia), Conservative Model (e.g. Germany, France, and Italy), and Social Democratic Model (e.g. Scandinavian countries), addressing how welfare provisions of health, housing, employment, pension, education etc. vastly differ among countries –the kinds of programs/services evaluators face all the time.

Hot Tip- Hot ‘Book’ 3: Three Cups of Tea by Greg Mortenson nicely exemplifies how it takes one man to promote peace by building schools in the midst of political and social turmoil at a remote distance from home-sweat-home. The author will inspire evaluators to gain practical skills, and pursue their goals no matter what in order to make the world a better place for all.

The American Evaluation Association is celebrating Minnesota Evaluation Association (MN EA) Affiliate Week with our colleagues in the MNEA AEA Affiliate. The contributions all this week to aea365 come from our MNEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Older posts >>

Archives

To top