My “Home-Made” Evaluation Tools to Avoid Gender (and Equity) Blindness by Sara Vaca

Hey there! I’m Sara Vaca (evaluation consultant) and here is my monthly Saturday post. I want to share how I try to introduce Gender perspective in every evaluation (regardless of the Terms of Reference (ToRs) mentioning it or not).

Lessons Learned: I heard during my Masters in Evaluation that many evaluations are gender-blind, and that term got stuck in my head as something I would definitely try to avoid. But how? After having conducted numerous evaluations, here are the practical tools I use for now:

Hot Tip: Always introduce Gender and Equity as evaluation criteria. Even if it is not in the ToRs, or the other criteria include some gender questions, I suggest we make it a stand-out one, to make sure we honor it properly and we talk about it in at least a very specific section of the report. Clients have never said no, quite the opposite.

I start the evaluation, and the questions about Gender and Equity help me consistently ask about these issues (usually details about how the programme was differently responding to and affecting different groups – women, men, children, displaced or host populations, rural or urban, different ethnicities, religions, wealth groups, sexual orientation, abilities, etc.), mainstreamed in every method and with all the stakeholders.

Cool Trick: Then I use the only gender-specific tool I had used so far was what I called Gender Analysis, where I gather all the data collected and I summarize and analyze how the crisis (or the programme) differently affected women (and girls) and men (and boys). Here are examples:

Analysis of how men and women are affected by the decentralization
of Mother to Child HIV transmission prevention services in Equatorial Guinea

Gender analysis of Livelihoods projects in rural Iraq

Rad Resource: But it has always felt like a rudimentary approach, and I want to upgrade my Gender skillsets. So I was excited to see the new UN Women’s manual: Inclusive Systemic Evaluation for Gender equality, Environments and Marginalized voices (ISE4GEMs): A new approach for the SDG era.

I have not finished exploring it, but I have made this infographic summarizing some extra tools and frameworks:

(Here complete).

Lessons Learned:  Going through this very useful tool, I realized 3 things:

  1. The only tool I use (my so-called Gender analysis) is an adaptation of the first framework in the manual.
  2. The other tools, though interesting, seem to be more focused on situation analysis and identifying root problems than to serve as data collection or data analysis methods.
  3. One challenge for me is how to introduce out of the scope issues like inequity root causes into discussions without generating awkward situations (or worse, losing my contracts!). In my long-ish term goal of becoming a feminist evaluator who tries to use evaluation as a transformational tool instead of just technical, I’m still in search of practical tools or ways to contribute make my evaluations more impactful.

Comments, ideas and tips are always welcome :-).

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on theaea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

19 thoughts on “My “Home-Made” Evaluation Tools to Avoid Gender (and Equity) Blindness by Sara Vaca”

  1. Hi Sara,

    I am currently a student at Queen’s University doing my Master of Education and I am in the process of completing a course in evaluation. This is my first foray into the world of evaluation.
    I wanted to thank you for your article on evaluation tools as I hadn’t thought about these biases and stereotypes which can ultimately affect a program outcome. I think we are so conscious now with not offending people that we may forget that different perspectives might be affecting the outcome of a program. It’s important to look at our evaluation from all angles as to understand how different parts of a program might be affecting different people, or on the flip side how different people affect parts of a program.

    Thanks again!
    Jess

    1. I wonder with how the world is today in 2021 if you might choose to involve other genders or identifiers in your evaluations?

      Is it also possible, when evaluating a program, to use a similar tool to avoid racial biases?

      Thanks again,

  2. Dear Sara,

    I am currently engaged in a graduate school program, taking a course in program evaluation. I am a little embarrassed to mention that up until roughly ten weeks ago I didn’t even have an inkling that program evaluation was a practice.

    Learning about program evaluation has truly kept me deep in the learning pit as I have navigated my way around a barrage of terminology, theory and new concepts. I wonder if it gets any easier? Within all the confusion and negotiation, the one theme that has propelled me out of the learning pit is the importance of the human factor that lies in every step of the evaluation process.

    I would like to thank you as I have read many of your blog entries and visited your website in which you have introduced me to the affects of data visualisation. Each of these continue to be helpful for my developing understanding. While completing a program evaluation design of my own for the very first time, of particular use and interest was your article, with Lovely Dhillon, on Refining Theories of Change in which you both address an alternate representation for a Theory of Change with causal mechanisms ‘above’ and ‘below’ the surface. I also appreciated your blog regarding the application of Evaluation Matrices as an effective means to collect, organise and track data using various methods. Most crucially, one of your blog entries that has left a profound impact on me is: My “Home-Made” Evaluation Tools to Avoid Gender (and Equity) Blindness which you posted on March 2, 2019. I hadn’t even considered the different impacts that developmental interventions can have on men (boys) and women (girls) and your collection of Gender Methods and Tools for ISE4EMS for gender analysis is eye-opening. You lay bare in your analysis how differently ‘men and women are affected by the decentralization of Mother to Child HIV transmission prevention services in Equatorial Guinea’. This is when program evaluation comes alive for me.

    I wonder how an evaluator is able to analyse data with sensitivity and without any implicit or unconscious gender bias? How does an evaluator navigate the political sensitivities around the collection of such data or maintain a lack of judgement in the face of gender inequality and/or an imbalance of gendered power relations, perceived or otherwise, that exists in different cultural and political contexts? How might an evaluator practise self-care ‘on the ground’ when positioned within traumatic or socially unjust contexts? How might an evaluator remain neutral when faced with situations or questions that might require them to share their own personal views, stories or perspectives?

    Sara, I am very interested to know how you first became engaged or interested in program evaluation and what your first experiences were in a professional and international context? The challenge of operating evaluations in different international, cultural and political contexts is something I have an interest in.

    I would like to humbly thank you again for engaging me in developing my understanding of program evaluation.

    Jamie Caton

  3. Hello,
    I was very interested in your article on gender and equity blindness. Being a self proclaimed strong women with a daughter who is equally as strong I am very aware of situations that do not seem to meet the needs of women as well as they do men. Working in a school I notice many of these events taking place daily. The sad part about this is that most equity issues are not done consciously, it is the unconscious bias that happens without awareness. Unless staff actively collect and analyze data collaboratively being intentional to look for inequities in gender, race, socio-economic areas etc… we fail to see them. It is often hard to admit that these equity issues still exist, and often will be immediately addressed by those who chose to look for them I am interested if there is an evaluation protocol that you are aware of that can easily be utilized by schools?

    I feel that I see more of the equity issues happening within economic bias where I work. Where students abilities or accessibility to interventions are limited because staff may believe that if parents aren’t making themselves available to help at home that there is no sense in the school providing consistent interventions. Or ski trips are planned yearly that are not an option for our students whose families have lower economic situations. Or books are not sent home because of the fear that they may not come back, maybe there are preconceived beliefs that students with not any lunch or not very good lunches have mothers who “don’t care”. I appreciate you writing about this in your blog, it reiterates the importance of being intentional about looking at equity.

    1. Thanks Cathy for your interesting reflections.
      Since my work is often done fast (as external evaluator in a “short” visit for final evaluations often), I focus more in finding very blatant issues and trying to induce this type of thinking into the organization’s staff, rather than start having more detailed, deeper assessment of the whole as you as internal evaluator can… All those issues you already identified seemed to be spot on on what we are discussing – and I guess just practice and keeping the “equity glasses” on will make us realize remaining blind spots (probably still many).
      Thanks.

  4. I enjoyed reading Sara Vaca’s discussion of evaluation tools and reporting formats for gender-related observations. However, reading further, I wondered whether there can be such a thing as a “feminist evaluator”–or is this an oxymoron? While I am a passionate advocate of my own and other women’s right to self-determination, and I have generated my share of awkward situations in advocating my position, it has not been in my role as an evaluator.

    I think of evaluators as filling a descriptive rather than a prescriptive role in knowledge management. I would not be surprised that the author’s discussions of “inequality root causes” has generated awkward situations if she spoke as an advocate rather than as a reporter. “Impactful” findings should be the result of careful and precise data collection rather than the desire to instigate social change.

    1. Thanks Gwenn. This is in fact a deep, key debate. I totally understand your point (as a matter of fact that is my present attitude towards it), but I am intrigued and fascinated by the transformative paradigm and experienced authors (such as Donna Mertens and many others) who say (not literally) that if evaluation is not used to change reality then is “part of the problem”… To be continued. Thanks!

      1. I don’t agree with Gwenn especially. Depending on the terms of the evaluation, it makes a great deal of sense to look at whether the potential program outcomes and impacts were reduced because of a failure to consider differentially the roles of males and females in the project. You can look at this from either a gender-conforming or a gender-transformational perspective.

        1. I concur that program participants’ gender is a relevant variable in determining program impacts. Sara and I were discussing how or whether the evaluator’s gender comes into play in the evaluative process.

          1. Gwenn, Certainly, participants’ sex is a relevant variable. But I am referring to the manner in which the project, program, or study determined in advance the potential impact of gender roles. This article provides a good example of what I mean: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5120030/
            And of course this is a situation where the program designers did not even look at gender roles from a gender-conforming way. As you will see, this article calls out this lack in the context of an evaluation. This is what I mean.

          2. Just after reading your message, I saw this article on a female anthropologist’s experiences with toilet facilities in rural locations around the world. Note her comments on how these constraints impact a researcher’s (or evaluator’s) choice of projects.
            Froystad, K. Failing the third toilet test: Reflections on fieldwork, gender, and Indian loos. Ethnography. https://doi.org/10.1177/1466138118804262

  5. Thank you for this. I am interested to know about the tools the evaluation community uses for On-grid power projects, both renewable and non-renewable.

    Thanks

      1. I’m afraid I’m not familiar with that type of projects, so I invite you to write a post about them?!
        Thanks.

        1. Thank you. Sorry, I did not see this earlier or I am not sure I replied. On-grid facilities are unique in the sense that once the power generated reaches the grid it becomes indistinguishable on who benefits from it. Even the report, you have kindly shared, talks more about off-grid and microgrid services than on-grid services. Hence the question. Regards. Rex

Leave a Reply to Sara Vaca Cancel Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.