AEA365 | A Tip-a-Day by and for Evaluators

TAG | equity

Our names are Susan M. Wolfe and Kyrah K. Brown and we are consultants at CNM Connect where we provide evaluation and capacity building services to nonprofit organizations.  Our work also includes evaluating community collaborations and coalitions. To effectively address most health, education, and other social issues at a systems level requires that communities address inequity and injustice.

RAD RESOURCE: In January, 2017 an article titled “Collaborating for Equity and Justice: Moving Beyond Collective Impact” was published in the Nonprofit Quarterly.

The authors presented the following six principles to promote equity and justice and each has implications for how coalitions and community collaboratives are evaluated.

Principle 1: Explicitly address issues of social and economic injustice and structural racism.

  • HOT TIP: Nearly all human problems, especially where there are disparities in outcomes, can be traced to social and economic injustice and/or structural racism. As an evaluator, examine whether these issues are being discussed and directly addressed.

Principle 2: Employ a community development approach in which residents have equal power in determining the coalition’s or collaborative’s agenda and resource allocation.

  • HOT TIP: Ask who has the actual power to make decisions and set agendas for the collaborative.

Principle 3: Employ community organizing as an intentional strategy and as part of the process. Work to build resident leadership and power.

  • HOT TIP: Closely examine the membership and leadership to determine the extent to which residents, or those who are directly affected by the issue at hand, are members and leaders.

Principle 4: Focus on policy, systems, and structural change.

  • HOT TIP: Review the agendas and activities to determine whether they are promoting more programs, or facilitating change in policies, systems, and structures.

Principle 5: Build on the extensive community-engaged scholarship and research over the last four decades that show what works, that acknowledge the complexities, and that evaluate appropriately.

Principle 6: Construct core functions for the collaborative based on equity and justice that provide basic facilitating structures and build member ownership and leadership.

RAD RESOURCE: The Collaborating for Equity and Justice Toolkit provided by The Community Tool Box can be accessed at: https://www.myctb.org/wst/CEJ/Pages/home.aspx

HOT TIP: Many nonprofits and health agencies are engaged in collaborative work and are oftentimes looking for effective frameworks to model after. Evaluators can use the Collaborating for Equity and Justice Toolkit to facilitate discussions and coalition development and planning efforts. When you introduce nonprofits and collaboratives to the framework, it may be helpful to provide brief presentations or facilitate interactive planning sessions. Prepare guided questions that help nonprofits to think about the application of the six principles in their work. As mentioned, the application of this framework can prove to be useful in refining or developing coalition goals that are intentional and evaluating their efficiency and effectiveness in meeting those goals.

The American Evaluation Association is celebrating Community Psychology TIG Week with our colleagues in the CP AEA Topical Interest Group. The contributions all this week to aea365 come from our CP TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi!  This is Michael Bamberger, an Independent Consultant specializing in the evaluation of social development and gender programs.  Over the past few years I have worked with United Nations organizations, bi-lateral programs and NGOs helping strengthen their evaluations of their gender equity policies.

Most international development agencies have now defined the promotion of gender equality (or gender equity) as one of their development goals, and most conduct periodic evaluations to assess the extent to which their programs and policies contribute to strengthening the economic, social and political empowerment of women and the reduction of the differences between women and men on these dimensions.  However, many of these evaluations tend to over-estimate the positive effects of their programs on promoting gender equality and frequently under-estimate or even ignore some of the negative outcomes of these programs. Frequently the evaluations only interview the women participating in the project, and produce glowing reports on the significant benefits, but fail to talk to women who did not participate.  The failure to identify negative outcomes is unfortunately very common and has serious implications.

Hot Tips:

  • Build into the Terms of Reference for the evaluation a requirement that the evaluators, even when working on a limited budget and under time-constraints, must identify and interview women (and where appropriate men) who did not benefit from the project.
  • Assess carefully the evaluation methodology to ensure that it is capable of identifying unintended outcomes.  Many evaluation methodologies such as results-based evaluations, many theories of change, and most experimental and quasi-experimental designs only measure the extent to which intended results have been achieved and are not able to capture unintended outcomes.
  • Be aware that many evaluations only obtain information from people directly involved in the program, most of whom will be reluctant to criticize the program which pays their salary.
  • In most communities there are people who are familiar with the program being evaluated and its effects, but who are not directly involved and who are able to provide a more balanced perspective.  Examples include: the district nurse (who usually knows almost all of the women in the community), the police chief (a useful source in cases of domestic violence), women’s organizations, school teachers and local religious leaders.
  • Use a mixed methods design that combines quantitative and qualitative data and that emphasizes the importance of triangulation to increase validity by systematically comparing information collected from different sources.

Rad Resources:  The World Bank “Gender and Transport” website illustrates many of the challenges and unintended consequences for women and girls of transport initiates which are assumed to be “gender neutral”.  Module 2 identifies the challenges and Module 5 identifies tools, including research tools, for promoting gender equity in this sector.

The American Evaluation Association is celebrating Feminist Issues in Evaluation (FIE) TIG Week with our colleagues in the FIE Topical Interest Group. The contributions all this week to aea365 come from our FIE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi, I’m Katherine Hay. I’ve spent the last 20 years in India working on development, research, and evaluation.

Lessons Learned: One thing I’ve learned, and say all the time is, “equity is not an intervention.”

In societies where equity is the driving goal, perhaps fairly straightforward evaluations of interventions can increase equity. In such societies evaluations could identify the ‘best’ programs, where ‘best’ is defined as reducing inequities. Armed with that knowledge we would design more of the ‘right programs’ and equity would be achieved.

But these societies do not exist. Most of the world is characterized by increasing inequity and development models that put equity on the back seat.

So how can evaluation increase equity in the real world? Is it reasonable to expect that interventions generated from systems that perpetuate gender and other inequities will lead to equity or that evaluation of interventions will deepen equity?

I practice evaluation because I think it is reasonable, but only if such evaluations are understood as intentional disruptions to inequitable systems. This entails seeing equity as emerging from and integral to movements to change societies rather than from technical tinkering within existing systems. This is why applying a feminist lens to evaluation is so core to the way I practice evaluation.

At worst, evaluation can reinforce inequities; on average it might reflect them, but at best it can challenge them. Feminist evaluation offers a lens that fosters designs, approaches, and tools which bring inequity to the foreground.

Hot Tips: I’m often asked, ‘how do you do feminist evaluation?’ There isn’t a simple checklist. The only way is by applying feminist principles at each stage of an evaluation.

Reach out and get involved. I’ve come to feminist evaluation by working with social activists, researchers and evaluators. We share designs, instruments, processes and challenges. I give time to feminist NGO’s with limited resources, but a lot of desire, to use evaluation to guide their work. Being part of these groups deepens my practice and experience. Find peers to challenge and inspire you.

Rad Resources: A few of us formed a gender and evaluation group that now has 646 members from around the world. Why not join the discussion?

Try to make feminist evaluation relevant to issues on the ground. Following a spate of brutal attacks on women in India, I changed a planned keynote at the last minute to discuss how evaluation can help end violence against women. It was a risk I’m glad I took. You can see the video here.

EvalPartners gives small grants to voluntary evaluation organizations to implement peer-to-peer, teaching, and evaluation advocacy projects. All proposals need to include equity and gender but it can also be the focus.

The American Evaluation Association is celebrating Feminist Issues in Evaluation (FIE) TIG Week with our colleagues in the FIE Topical Interest Group. The contributions all this week to aea365 come from our FIE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

I’m Chad Green, Program Analyst at Loudoun County Public Schools in Northern Virginia, and Chair of the PreK-12 Educational Evaluation TIG. Welcome to our sponsored AEA365 week in conjunction with Teacher Appreciation Week, which is celebrated from May 5-9 in the U.S.

Since 2011, I have championed our TIG’s core values below as quoted from our website.

“The PreK-12 Educational Evaluation TIG values relevant, responsive, high quality educational evaluation that reflects our beliefs in social justice, equity, and the importance of educating the whole child.”

How so? When the language and actions of my colleagues and I align with these core values, I acknowledge and celebrate them publicly. Barbara Taylor called this form of strategic dialogue “metasensemaking” in her study of elementary school principals involved in the process of organizational change. She defined it as “a form of organizational enactment used to further the potential for organizational momentum and individual motivation.” The “Walking the Talk” section of the monthly AEA Newsletter is another example of this leadership practice.

The beauty of our TIG’s values is that, as essentially contested concepts, anyone can take an active role in their interpretation and enactment. While I have had the opportunity to celebrate instances of equity and whole child education in action over the years, social justice has eluded me. I attribute this shortcoming primarily to my own lack of understanding of this fuzzy concept.

Lesson Learned: I only write this now because recently I have learned that, during a persuasive presentation by Paul Carr of Lakehead University in Canada, many people understand the role of democracy in education only in a “thin” way.

Last month a few of us from the TIG’s leadership team attended the Annual Meeting of the American Educational Research Association. In his panel presentation sponsored by the Dewey Studies SIG, Carr introduced the following framework of teacher/student engagement in democracy which builds on the seminal work of Benjamin Barber.

Carr's notion of Thin vs. Thick Democracy

Carr’s notion of Thin vs. Thick Democracy

According to Carr, a thin connection to democracy in education is exemplified by weak linkages between the school experiences of teachers and students and the broader experiences of society in general.  He argued that social justice can only be expressed on the thick end of the spectrum, in which freer forms of democracy influence all aspects of how education is organized (e.g., academic curriculum, assemblies, extra-curricular events, staff meetings, etc.).

Carr’s presentation struck a chord because it reminded me of Sandra Mathison’s panel presentation last year on her efforts to bridge the evaluation culture gap in school systems throughout British Columbia (see below).  I look forward to celebrating other educational evaluators who have integrated social justice into their practice.

Mathison's notion of the Evaluation Culture Gap

Mathison’s notion of the Evaluation Culture Gap

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Hello, I’m Jori Hall, assistant professor at the University of Georgia and a member of the AEA Public Statement on Cultural Competence in Evaluation Dissemination Working Group. This tip is focused on integrating cultural competence into everyday practice through values-engagement.

Tips:

  • As suggested in the AEA Public Statement on Cultural Competence in Evaluation, all evaluation practice and evaluands are situated in and influenced by cultural norms, values, and various ways of knowing. Values-engagement acknowledges these influences and attempts to be responsive to the dynamic interaction between the values reflected in evaluation practice and the evaluand. That is, values-engaged evaluators understand that evaluation practice promotes values, and that these values must respectfully engage stakeholders’ values.
  • Values-engagement is not a specific strategy or a set of required methods; rather, it is a commitment to culturally responsive evaluation. While there is more than one way to be values-engaged, the commitment to culturally responsive, values-engagement suggested here involves the evaluator prioritizing values of inclusion and equity in everyday practice. Inclusion refers to engaging and describing the plurality of stakeholders’ values, perspectives, and concerns, focusing on the least well served in a particular context. Equity refers to how well and to what extent the evaluand is attending to stakeholder groups (i.e., access, participation, etc.) in the context. Because values-engagement advocates inclusiveness and the equitable treatment of stakeholders, it supports the goals of the Statement on Cultural Competence in Evaluation.
  • Values-engagement can be integrated throughout the life cycle of an evaluation, and enacted through generating evaluation questions, data, and dialogues related to the ways in which the evaluand is attending to the cultural values of the groups represented in the context. To learn more about values-engagement, its connection to cultural competence, and how evaluators can practically enact its commitments in different evaluation contexts, begin with the resources provided below!

Rad resources:

The American Evaluation Association will be celebrating Cultural Competence Week. The contributions all this week come from the Cultural Competence committee. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · · · · · · ·

Archives

To top