AEA365 | A Tip-a-Day by and for Evaluators

TAG | vulnerable populations

Hello from Mary Crave and Kerry Zaleski, of the University of Wisconsin – Extension and Tererai Trent of Tinogona Foundation and Drexel University.  For the past few years we’ve teamed up to teach hands-on professional development workshops at AEA conferences on participatory methods for engaging vulnerable and historically under-represented persons in monitoring and evaluation. Our workshops are based on:

  • More than 65 years of collective community-based experience in the US and more than 55 countries
  • Our philosophy that special efforts should be made to engage people who have often been left out of the community decision-making process (including program assessment and evaluation)
  • The thoughtful work of such theorists and practitioners as Robert Chambers, a pioneer in Participatory Rural Appraisal.

Lessons Learned: While many evaluators espouse the benefits of participatory methods, engaging under-represented persons often calls for particular tools, methods and approaches. Here’s the difference:

  1. Vulnerability: Poverty, cultural traditions, natural disasters, illness and disease, disabilities, human rights abuses, a lack of access to resources or services, and other factors can make people vulnerable in some contexts. This can lead to marginalization or oppression by those with power, and critical voices are left out of the evaluation process.
  2. Methods and tools have many benefits: They can be used throughout the program cycle; are adaptable to fit any context; promote inclusion, diversity and equality; spark collective action; and, support community ownership of results – among others.
  3. 3.     Evaluators are really facilitators and participants become the evaluators of their own realities.

Hot Tip:  Join us to learn more about the foundations of and some specific “how-to” methods on this topic at an upcoming AEA eStudy, February 5 and February 12, 1-2:30 PM EST. Click here to register.

We’ll talk about the foundations of participatory methods and walk through several tools such as community mapping, daily calendars, pair-wise ranking, and pocket-chart voting.

Rad Resources: Robert Chambers 2002 book: Participatory Workshops: A Sourcebook of 21 Sets of Ideas and Activities.

Food and Agricultural Organization (FAO) of the UN: http://www.fao.org/docrep/006/ad424e/ad424e03.htm (click on publications, type in PLA in search menu)

AEA Coffee Break Webinar 166: Pocket-Chart Voting-Engaging vulnerable voices in program evaluation with Kerry Zaleski, December 12, 2013 (recording available free to AEA members).

June Gothberg on Involving Vulnerable Populations in Evaluation and Research, August 23, 2013

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Greetings, I am June Gothberg, incoming Director of the Michigan Transition Outcomes Project and past co-chair of the Disabilities and Other Vulnerable Populations topical interest group at AEA.  I hope you’ve enjoyed a great week of information specific to projects involving these populations.  As a wrap up I thought I’d end with broad information on involving vulnerable populations in your evaluation and research projects.

Lessons Learned: Definition of “vulnerable population”

  • The TIGs big ah-ha.  When I came in as TIG co-chair, I conducted a content analysis of the presentations of our TIG for the past 25 years.  We had a big ah-ha when we realized what and who is identified as “vulnerable populations”.  The list included:
    • Abused
    • Abusers
    • Chronically ill
    • Culturally different
    • Economically disadvantaged
    • Educationally disadvantaged
    • Elderly
    • Foster care
    • Homeless
    • Illiterate
    • Indigenous
    • Mentally ill
    • Migrants
    • Minorities
    • People with disabilities
    • Prisoners
    • Second language
    • Veterans – “wounded warriors”
  • Determining vulnerability.  The University of South Florida provides the following to determine vulnerability in research:
    • Any individual that due to conditions, either acute or chronic, who has his/her ability to make fully informed decisions for him/herself diminished can be considered vulnerable.
    • Any population that due to circumstances, may be vulnerable to coercion or undue influence to participate in research projects.

vulnerable

Hot Tips:  Considerations for including vulnerable populations.

  • Procedures.  Use procedures to protect and honor participant rights.
  • Protection.  Use procedures to minimize the possibility of participant coercion or undue influence.
  • Accommodation.  Prior to start, make sure to determine and disseminate how participants will be accommodated in regards to recruitment, informed consent, protocols and questions asked, retention, and research procedures including those with literacy, communication, and second language needs.
  • Risk.  Minimize any unnecessary risk to participation.

Hot Tips:  When your study is targeted at vulnerable populations.

  • Use members of targeted group to recruit and retain subjects.
  • Collaborate with community programs and gatekeepers to share resources and information.
  • Know the formal and informal community.
  • Examine cultural beliefs, norms, and values.
  • Disseminate materials and results in an appropriate manner for the participant population.

Rad Resources:

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

Hi, I am Robin T. Kelley and am an internal evaluator at a national nonprofit health organization that is funded by the Centers for Disease Control and Prevention to provide free capacity building assistance to HIV prevention organizations, health departments and their HIV planning groups.

In the HIV/AIDS field, there are a number of changes occurring; here are just a few major ones:  In 2010, there was the release of the U.S. National HIV/AIDS Strategy. All funded entities are now striving to align themselves with the major goals of this strategy. As of  2011, scientific studies that showed the effectiveness of adherence to HIV medicine in reducing the viral loads,  resources are placed into, biomedical interventions and the  emphasis is now placed  more on organizations conducting  high impact HIV prevention.

Lessons Learned:

One key method of building an organization’s ability to manage complex situations, particularly small organizations that serve vulnerable populations, or populations of color-is to strengthen their change management leadership skills.  Research has shown that in times of complexity, such as shifting federal and health priorities, organizations, businesses that serve minorities  often shut their doors first ,leaving underserved communities abandoned and without services.  To sustain these agencies, evaluators as well as program managers should be agile and flexible in understanding the community needs, their resources, staff strengths as well as weaknesses-to best manage the changes.

Hot Tips:

Here are some steps to take and useful tools to address HIV changes and changes in general:

1)     First, help the organization conduct an organizational diagnosis.  They must know what they have in order to consider what to change.

2)     Second, help the organization to conduct an environmental scan or asset mapping of their community to determine if there is still a need for their services.

3)     Then to help organizations to analyze the data.  Based on the findings, help the organization to do a SWOT analysis (an analysis of their strengths, weaknesses, opportunities and threats).  Depending on these findings,  perhaps  there is a way to merge efforts with another organization;

4)     Next, help the organizations communicate changes to all staff; without constant communication, rumors can fly and morale can sink.

5)     Finally, help the organization to create a process log so that they can record the number of new service requests and activities and to continue to justify their existence.

Rad Resources:

The American Evaluation Association is celebrating the Business, Leadership, and Performance TIG (BLP) Week. The contributions all week come from BLP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello! We’re Allan Porowski from ICF International and Heather Clawson from Communities In Schools (CIS). We completed a five-year, comprehensive, mixed-method evaluation of CIS, which featured a several study components – including three student-level randomized controlled trials; a school-level quasi-experimental study; eight case studies; a natural variation study to identify what factors distinguished the most successful CIS sites from others; and a benchmarking study to identify what lessons CIS could draw from other youth-serving organizations.  We learned a lot over the years, and wanted to share a few big takeaways with you about conducting evaluations on interventions for at-risk youth.

Lessons Learned:

  • Sometimes, you have to catch falling knives: We found that the students coming into CIS were targeted for services because they were on the strongest downward trajectories on a number of factors (e.g., academics, behavior, family issues, attendance, etc.). There’s an old adage in stock market trading that you should “never catch a falling knife” – but that’s what CIS and other dropout prevention programs do every day. This has implications for how you evaluate the relationship between dosage and outcomes. A negative relationship between dosage and outcomes doesn’t necessarily indicate that services aren’t working – it can actually be an indication that services are going to where they are needed the most.
  • Look for the “Nike Swoosh”: The general pattern of outcomes among CIS students looked like Nike’s “swoosh” logo: There was an initial downward slide followed by a longer, more protracted period of improvement. Reversing that initial downward slide takes time, and this pattern is worth investigating if you’re evaluating programs for at-risk youth.
  • As the prescient rock band Guns n’ Roses put it, “All we need is just a little patience”: Needless to say, it takes a long time to turn a child’s life around. So many evaluations of at-risk students don’t have a long enough time horizon to show improvements, which may in part explain why we see such low effect sizes in dropout prevention research relative to other fields of study.

Rad Resources:

  • Executive Summaryof Communities In School’s Five-year National Evaluation
    • Communities In Schools has great ideas and resources for dealing with at-risk youth. CIS surrounds students with a community of support, empowering them to stay in school and achieve in life. Through a school-based coordinator, CIS connects students and their families to critical community resources, tailored to local needs. Working in nearly 2,700 schools, in the most challenged communities in 25 states and the District of Columbia, CIS serves nearly 1.26 million young people and their families every year.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Archives

To top