AEA365 | A Tip-a-Day by and for Evaluators

TAG | gender

We are Silvia Salinas-Mulder, Bolivian anthropologist, feminist activist and independent consultant, and Fabiola Amariles, Colombian economist, founder and director of Learning for Impact. We have worked for several years as external evaluators for development programs in Latin America. The following ideas may help to operationalize the principles of gender- and human rights (HR)-responsive evaluation in complex, multicultural contexts.

Lesson learned: Terms of Reference (TOR) for an evaluation are not engraved in stone.

Tip: Reframe the often conventional evaluation questions and other aspects of the evaluation process to ensure that gender and HR issues surface, and evidence of change (or no change) in women’s lives is gathered. Take into account context-specific issues and gender dynamics, as well as relevant cultural patterns, such as the effects of migration in the family roles and decision-making processes within some agricultural community settings.

Lesson learned: Some stakeholders are tired of being interviewed, while others – especially rural women- are eager to be heard.

Tip: Be creative; evaluation techniques are the means not the end, and can thus permanently be created, recreated and adapted to each situation and context. For example, use “conversatorios” (round table discussions), as opposed to focus groups, to gather people with diverse backgrounds and perspectives to discuss over a particular issue of the evaluation; participants usually appreciate these reflective spaces and feel motivated to speak “outside the box”, while evaluators take a holistic overview of the topic.  Drawings, role plays and other popular education techniques may also facilitate participation of marginalized groups, including illiterate women.

Lesson learned: Answers to your questions may not contain key gender and HR issues to understand how change is occurring.

Tip: Awareness of specific cultural and gender communication patterns is crucial for an effective exchange. In any case, interviews should be dealt with as dialogues where people have the opportunity to express their priorities and points of view. Do not limit your interactions to a question-answer dynamic. Let people speak freely and “listen actively” to discover the essential. Respect and interpret the silences and do not insist on answers to your questions, rather focus on trying to understand the underlying meaning of each reaction. This will allow an eventual reconstruction of how change is occurring (Theory of Change) for the specific intervention and context, even if it has not been explicitly stated in the project design. Also, as evaluators we tend to focus on verbal communication, ignoring the importance of tone and gestures. Make sure you are alert to less explicit key messages.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I am Lillian Rivera, After School Program Director for Hetrick-Martin Institute, a community-based organization that provides programs for LGBT teens that focus on arts and culture, health and wellness, job readiness, academic enrichment as well as counseling and HIV prevention services to 1,500 youth annually.  HMI’s HIV prevention work allows us to work with various evaluators using different quantitative and qualitative assessments.

I’m writing to share a field perspective on the issue of adding LGBTQ questions to these assessments.  The AEA365 post from John Daws in February notes “’Are you Male or Female?’ That simple question might be adequate sometimes, but a good evaluator should realize when it isn’t.”  Evaluating programs that serve HMI’s population certainly demands that the question of gender be approached with thinking beyond the traditional binary of male and female, but often instruments don’t allow that.

Even instruments  with choices for male, female, and transgendered, which seem inclusive and accurate, allowing survey takers to “see themselves in the data” and the evaluation team to learn about the population served, can limit the accuracy of reflected demographics.

Lesson Learned: Since LGBTQ populations, particularly youth, may be at various states in their gender identity transition at different survey time points respondents frequently connect to different gender labels at different times and therefore may select different answers for each.  This can yield inaccurate information and limit the ability to interpret findings.

Lesson Learned: A great solution is to use a secondary measure on which program or site staff confidentially indicate which participants are gender variant, individuals whose gender identity may vary.  Triangulating staff interview data, clinical intake statistics on participants, and initial demographics findings from the participant survey let the evaluation team identify the survey’s inaccuracy.  Meeting with program staff yielded the use of the secondary measure to give accurate data while respecting participant process and confidentiality issues.

Bonus: Having staff help generate the solution created buy-in to the use of the measure and the evaluation process overall.

Hot Tip: When instrument and observational reflections seem at odds, work with practitioners to find innovative measurement solutions.  Don’t be afraid to think outside of the box—or the gender binary!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

I’m Susan Kistler, the American Evaluation Association’s Executive Director, and I contribute each Saturday’s aea365 post. In last week’s post, I told you how to win a copy of Evaluation Strategies for Communicating and Reporting by suggesting your favorite (or a hated) data visualization via the aea365 comments.

Hot Tip: We received lots of great entries. Check out the comments from that post to get ideas for the dos and don’ts of data visualization, and to see who won!

This week, I’ve enjoyed learning from the folks in the LGBT TIG here on aea365 as well as via Joe Heimlich’s fantastic webinar this past Thursday on considerations when including questions of gender and sexuality as part of evaluation. If you are an AEA member, you can see the recording of the webinar, free in the AEA webinars archive.

Last week’s drawing went so well, and our LGBT TIG leaders were enthusiastic and inclined, that we’re trying another this week.

Cool Trick: Sponsored by our colleagues in the LGBT TIG, enter this week’s aea365 drawing! To enter, all you have to do, by midnight on Thursday, February 24, in any time zone, is to add a note to the comments section of this post that gives your take on when is a time that you should, or should not, include sexuality or gender identify in an evaluation?

We’re not looking for a right answer, but rather your thoughts on the question.

As with our previous drawing, you don’t have to be a member of AEA (but if you aren’t, wouldn’t you like to join?) to enter or win. If you are receiving this via email, just click on the post’s title to return to this post on the aea365 website and scroll down and add a comment.

One entry per person please (although you are welcome to comment on each other’s suggestions all you wish)! We will randomly select a winner from all those who add an on-topic comment. The winner will receive a copy of Finding Out: An Introduction to LGBT Studies generously donated by our colleagues at SAGE publications.

The American Evaluation Association is celebrating LGBT Week with our colleagues in the LGBT Evaluation Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · ·

Greetings! I’m Joe Heimlich, a Professor at The Ohio State University with Extension, the School of Environment & Natural Resources, and the Environmental Science Graduate Program. I’m also a Senior Research Associate with the Institute for Learning Innovation. This year, along with John Daws, I’m Program co-Chair for the AEA LGBT Issues TIG.

Hot Tip: One of the most ethical decisions we make as evaluators, is determining what we want to know about a person. Age, income, race, ethnicity, educational level…all carry with them ethical questions about use, need, confidentiality and more. Often, there are just reasons for deciding to ask LGBTQQI questions. (That unpronounceable acronym stands for lesbian, gay, bisexual, transgender, queer, questioning, and intersex. Often it’s shortened to LGBT.)

There are two major conditions shaping the decision to include – or to omit intentionally – questions on sexual or gender identity, and neither relates to LGBT politics:

  1. When such data would further our understanding of the effect or the impact of a program, treatment, or event. The rule of thumb I use is that if I am making assumptions using sex (Male or Female, as biological indicators) that involve gender role issues, then I need to include gender identity as a factor. This is especially true given research findings that in many situations, gay men are more like straight women in some decision and interaction processes. My favorite example is the question of dark versus milk chocolate preference. And yes, in groups straight men do eat milk more than dark chocolate, but the straight women and the gay men eat dark more than milk. Nature? Nurture? That we don’t know suggests to me that we should ask more and assume less.
  2. When asking for such data would benefit the individual and/or their engagement in the evaluation process. We all like to be included, and the chance to “see” oneself is important. In my recent interviews of same-sex couples about museum membership, one example kept coming up: The museums’ membership forms did not allow for gay and lesbian households to self-identify. The unfriendliest forms had two lines to enter names, labeled Male and Female. Other forms allowed for two names to be entered, but did not allow them to indicate what relationship existed between the two. If our evaluations are designed to allow people ‘s voices to be heard, there may be times when we need to let them know we want to hear their full voice, which means including all of who they are.

Rad Resource: A leading edge is the American Psychological Association’s Division 44, Society for the Psychological Study of Gay, Lesbian, Bisexual, and Transgender Issues. This is the group that first identified 11 genders.

The American Evaluation Association is celebrating LGBT Evaluation Week with our colleagues in the LGBT AEA Topical Interest Group. The contributions all this week to aea365 come from our LGBT members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting LGBT resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · · ·

My name is Denice Cassaro. I am strongly committed to serving as a positive social change agent and educator through the use of evaluation processes. I have found incorporating queer/feminist/critical race theories helpful by the challenges offered in:

  • understanding concepts of identities (race/ethnicity, sex/gender, sexual orientation/preference) as socially constructed
  • not minimizing the impact/meaning of intersecting identities (experiences of someone who identifies as an African-American gay man vs as a heterosexual Latina)

In evaluation, an area that most frequently engages directly with issues of identity is in the collection of “demographic data.” I would like to explore with you considerations when forming identity categories.

Lesson Learned: Our choices of categories, headings, and response options, all have implications in terms of respondents’ feelings of inclusion or exclusion, the parameters for interpretation of results, and the parameters of possible effects from the interpreted results.

Hot Tip: Use open-ended demographic categories rather than those with limited options based on coding “efficiency”. Good examples include:

  1. When asked about your racial or ethnic identity(ies), how do you identify yourself?
  2. What racial or ethnic identity(ies) do others who do not know you attribute to you?
  3. When asked about your sexual orientation or preference, how do you identify yourself?
  4. What sexual orientation or preference do others who do not know you attribute to you?

Benefits include:

  • Feelings of inclusivity for participant
  • Awareness of scope of identities prior to coding allowing for a variety of coding methods
  • Certain identities may emerge as significant (in my work, being Jewish; identifying as LBGT ally)
  • Understanding for some identities, a person’s experiences may be different based on their “perceived” identity (in U.S., frequently true for some Latinos/Latinas who may be identified as African-American or white; being perceived as gay/lesbian when heterosexual or vice versa)
  • Insight into the impact certain identities have on individuals or groups (my work: “white, I really am a bad person”; “I am not black, I am Ghanaian American. I don’t like to be categorized as black, it’s just a color not who I am”; “I’m straight, I’m NOT gay”; “Straight, with GLBT sympathies”)

Rad Resource: To understand more about issues of gender, sexuality, and evaluation, consider reading New Directions for Evaluation Number 96 – Feminist Evaluation: Explorations and Experiences, Denise Seigart and Sharon Brisolara, editors.

Please note: for this discussion I am using the terms race/ethnicity, sex/gender and sexual orientation/preference though the combinations do not mean the same thing and their meanings have been conflated in daily use and in our practice. That is a discussion for another day!

The American Evaluation Association is celebrating LGBT Evaluation Week with our colleagues in the LGBT AEA Topical Interest Group. The contributions all this week to aea365 come from our LGBT members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting LGBT resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · · ·

My name is Käri Greene and I’m a Senior Research Analyst at an intergovernmental agency for the Oregon Public Health Division and Multnomah County Health Department, as well as a co-Chair for the LGBT Issues TIG. Our TIG explores areas of sexuality, gender and identity as they relate to evaluation theory, practice, and use, specifically focusing on issues related to lesbian, gay, bisexual, and transgender issues.

Many evaluations might not deal explicitly with LGBT issues; however, gender and sexuality are concepts present in much of our evaluation practice. Gender or ‘sex’ is a standard demographic variable, and sexual orientation is being included more frequently in evaluations. But the concepts of sexual orientation, sexual behavior, and gender identity can be dynamic and complex.

In public health evaluations, someone served by a program might identify as a lesbian woman, but she may have been born and raised as a boy and not identify as transgender. A man served at the local public health clinic might be having sex with other men, but not identify as gay or bisexual. Being clear about what we need to know about program stakeholders is essential to answering evaluation questions.

Lessons Learned – Question assumptions and ask the right questions for your evaluation and those served by the program: Sexual orientation does not automatically define a person’s sexual behavior, and gender identity does not always fit neatly into a two-by-two table.

Feeling confused about how to deal with gender and sexuality? That’s good – that means you’re questioning assumptions! The field is evolving and even after a century of research on sexuality and gender, few researchers agree on terminology, dimensions and categorical classifications of sexuality. But fear not, we’ll have more to say on this subject throughout the week…

Hot Tip: Consider how you currently assess gender. It might be important to ask multiple items to get at gender – one that asks current gender identity (“Do you consider yourself to be male, female, transgender, or something else?”) and one that asks birth gender (“What sex were you assigned at birth – male, female, or intersex?”).

Hot Tip: Consider expanding your existing response categories for sexual orientation. Younger clients might consider themselves “queer” but not the more traditional categories of lesbian, gay, or bisexual.

Rad Resource: The Sexual Minority Assessment Research Team (SMART), a multidisciplinary and multi-institutional collaboration, created a helpful document on best practices for asking about sexual orientation.

Rad Resource: The Human Rights Campaign has a helpful, simple list of terms on gender identity.

Rad Resource: The Gay & Lesbian Alliance Against Defamation (GLAAD) has resources, including a media reference guide, that can be helpful when communicating and reporting about issues of sexuality and gender.

The American Evaluation Association is celebrating LGBT Evaluation Week with our colleagues in the LGBT AEA Topical Interest Group. The contributions all this week to aea365 come from our LGBT members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting LGBT resources. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

· · ·

Archives

To top