AEA365 | A Tip-a-Day by and for Evaluators

TAG | Disabilities

Hi, we are Monika Mitra and Lauren Smith from the Disability, Health, and Employment Policy unit in the Center for Health Policy and Research at the University of Massachusetts Medical School.  Our research is focused on health disparities between people with and without disabilities.

Evaluating a Population of People with Disabilities

In collaboration with the Health and Disability Program (HDP) at the Massachusetts Department of Public Health (MDPH), we conducted a health needs assessment of people with disabilities in Massachusetts.  The needs assessment helped us better understand the unmet public health needs and priorities of people with disabilities living in MA.  We learned a tremendous amount in doing this assessment and wanted to share our many lessons learned with the AEA365 readership!

Lessons Learned:

  • 3-Pronged approach

Think about your population and how you can reach people who might be missed by more traditional methodologies:  In order to reach people with disabilities who may not be included in existing health surveys, we used two other approaches to complement data from the MA Behavioral Risk Factor Surveillance System (BRFSS).  They included: an anonymous online survey on the health needs of MA residents with disabilities and interviews with selected members of the MA disability community.

  • Leveraging Partnerships

Think about alternative ways to reach your intended population:  For the online survey, we decided on a snowball sampling method.  This method consists of identifying potential respondents who in turn identify other respondents; it is a particularly useful methodology in populations who are difficult to reach and may generally be excluded from traditional surveys and affect one’s generalizability of findings.  HDP’s Health and Disability Partnership provided a network to spread the survey to people with disabilities, caregivers, advocates, service providers, and friends/family of people with disabilities.

  • Accessibility is Key

Focus on accessibility:  In an effort to increase the accessibility of the survey, Jill Hatcher from DEAF, Inc. developed a captioned vlog (a type of video blog) to inform the Deaf, DeafBlind, Hard of Hearing, and Late-Deafened community about the survey.  In the vlog, she mentioned that anyone could call DEAF, Inc. through videophone if they wanted an English-to-ASL translation of the survey.  Individuals could also respond to the survey via telephone.

Rad Resources:

  • Disability and Health Data System (DHDS)

DHDS is an online tool developed by the CDC providing access to state-level health data about people with disabilities.

  • Health Needs Assessment of People with Disabilities Living in MA, 2013

To access the results of the above-mentioned needs assessment, please contact the Health and Disability Program at MDPH.

  • A Profile of Health Among Massachusetts Residents, 2011

This report published by the MDPH contains information on the health of people with disabilities in Massachusetts.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Greetings, I am June Gothberg, incoming Director of the Michigan Transition Outcomes Project and past co-chair of the Disabilities and Other Vulnerable Populations topical interest group at AEA.  I hope you’ve enjoyed a great week of information specific to projects involving these populations.  As a wrap up I thought I’d end with broad information on involving vulnerable populations in your evaluation and research projects.

Lessons Learned: Definition of “vulnerable population”

  • The TIGs big ah-ha.  When I came in as TIG co-chair, I conducted a content analysis of the presentations of our TIG for the past 25 years.  We had a big ah-ha when we realized what and who is identified as “vulnerable populations”.  The list included:
    • Abused
    • Abusers
    • Chronically ill
    • Culturally different
    • Economically disadvantaged
    • Educationally disadvantaged
    • Elderly
    • Foster care
    • Homeless
    • Illiterate
    • Indigenous
    • Mentally ill
    • Migrants
    • Minorities
    • People with disabilities
    • Prisoners
    • Second language
    • Veterans – “wounded warriors”
  • Determining vulnerability.  The University of South Florida provides the following to determine vulnerability in research:
    • Any individual that due to conditions, either acute or chronic, who has his/her ability to make fully informed decisions for him/herself diminished can be considered vulnerable.
    • Any population that due to circumstances, may be vulnerable to coercion or undue influence to participate in research projects.

vulnerable

Hot Tips:  Considerations for including vulnerable populations.

  • Procedures.  Use procedures to protect and honor participant rights.
  • Protection.  Use procedures to minimize the possibility of participant coercion or undue influence.
  • Accommodation.  Prior to start, make sure to determine and disseminate how participants will be accommodated in regards to recruitment, informed consent, protocols and questions asked, retention, and research procedures including those with literacy, communication, and second language needs.
  • Risk.  Minimize any unnecessary risk to participation.

Hot Tips:  When your study is targeted at vulnerable populations.

  • Use members of targeted group to recruit and retain subjects.
  • Collaborate with community programs and gatekeepers to share resources and information.
  • Know the formal and informal community.
  • Examine cultural beliefs, norms, and values.
  • Disseminate materials and results in an appropriate manner for the participant population.

Rad Resources:

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

Hello from Kansas, the nation’s breadbasket!  I am Linda Thurston, Associate Dean of the College of Education at Kansas State University and long-time member of AEA. I am the 2013 co-chair of AEA’s Disabilities and Other Vulnerable Populations (DOVP) TIG.  DOVP welcomes you to a week of aea365 articles focused on information and resources to help evaluators include vulnerable populations in their work.

Many evaluators are involved with K-12 education and the assessment or evaluation of teacher performance. To date, indicators of teacher quality have primarily been observations and student test scores.  Whether or not we, as evaluators, agree with this trend, we are always interested in assuring that our evaluation measures are valid.  If teacher evaluation systems do not acknowledge the presence of special populations of students there are grave concerns for validity and equity. In the May issue of Educational Researcher, Nathan Jones and his colleagues discuss the issues of including students with disabilities (SWD) and English language learners (ELL) in evaluating teacher performance. They also offer some suggestions that I think are applicable for many types of evaluations involving students with disabilities and other vulnerable populations.

Rad Resource: Article by Jones, Buzick, and Turkan, S. in the 42nd volume of Educational Researcher.

Despite advances in research on teacher evaluation (for  summaries, see Harris, 2011; Bell et al., 2012), there has been  virtually no attention given to whether teachers are effectively  educating exceptional populations—namely students with  disabilities (SWDs) and English learners (ELs).

Hot Tips:

  • For observing teacher performance in ways that include SWDs and ELLs, consider using protocols designed specifically for use with these special populations.
  • Assure that observers are trained in the instructional needs of both SWDs and ELLs.
  • In measuring student progress, examine and test assumptions about the presence of scores from SWDs and ELLs in general classroom settings (most SWDs and ELLs spend most of their time in general education classrooms).
  • Utilize a consistent system to consider use of accommodations and changes in classifications across time and to distinguish subgroups within both populations.

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

As an evaluator I am often confronted by questions that I often think about while taking the Staten Island Ferry at the end of the day. My name is Stan Capela and I am the VP for Quality Management and Corporate Compliance Officer for HeartShare Human Services. I am also the current AEA Government TIG Chair.

One of the many dilemmas an evaluator faces is the time to ponder questions that provide an opportunity to think about evaluation in a different way. Very often, these questions play a role on how I approach evaluation during the course of the day.

Lesson Learned: I have been an internal evaluator since 1978 where I worked for the first ten years at Catholic Charities, followed by 22 years at HeartShare. I have come in contact with a wide range of issues none more important than the impact a program evaluation has on the individuals served by the program. Often I ask myself a very simple question, what impact will my evaluation report have on the individuals served by the program? I raise this question more often since HeartShare serves a population that is predominantly developmentally disabled. If you are knowledgeable of such a population you know that a program’s impact on this population can often be incremental.

Lesson Learned: When you try to conduct consumer surveys focusing on developmentally disabled stakeholders you are often left with a choice of whether to attempt an interview with an individual whose ability to comprehend the question may be limited; rely on families who may have limited contact with the individual; or turn to staff who may fear if they answer the question honestly there may be a negative impact on their employment.

This leads to a very simple question around which I am seeking feedback. Specifically, as an internal evaluator can you care about the population that is served by the program and may be impacted by the results of your findings? If you care, will that have an effect on your ability to be objective in reporting your findings? Maybe I am the only one but as an internal evaluator who has devoted 32 years of his life to the field of evaluation I often have asked myself these questions. What about you? Let me know by forwarding your comments to stan.capela@heartShare.org or sharing them via the comments below.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Mary Moriarty, independent consultant and evaluator with Picker Engineering Program at Smith College. For 10 years I have specialized in evaluation of programs that serve underrepresented populations, particularly in science, technology, engineering, and mathematics (STEM). I previously directed several programs focused on increasing representation of individuals with disabilities in STEM.

I now realize the importance of ensuring cultural relevancy for effective project evaluation. Nowhere is this more critical than disability-based evaluations where contextual factors impact all phases of the evaluation. Here are some tips helpful in planning and implementing disability-based evaluations.

Hot Tip – Understand the Population: One of the most critical factors is determining impact on the populations being examined. However, in disability programs there can be significant disparities in definitions and classification systems. Some projects use definitions provided by the Americans with Disabilities Act others use internal or funding agency definitions. Comparing data becomes confusing or difficult, particularly when working with multiple agencies or programs. As evaluators we need to be aware of these differences so we can provide clarity and direction to the evaluation process.

Hot Tip – Understand the Impact of Differences: No two individuals with disabilities are alike; therefore evaluators need to understand the range and types of disabilities. Differences may present challenges on many fronts. First, developing comparison measures can be difficult when there are significant differences between individuals within the population. For example, the experience of an individual who uses a wheel chair may be different than that of an individual with a learning disability. Second, many individuals with disabilities have experienced some level of discrimination and may be reluctant to disclose sensitive information. There may be issues around confidentially or disclosure that could impact evaluation results. Being sensitive to these issues, establishing rapport, and utilizing a wide range of qualitative and quantitative measures will help to ensure the collection of accurate and useful data.

Hot Tip -Design Tools, Assessment Measures, and Surveys that are Universally Accessible: Third, we need to ensure that all evaluation methods and measures meet accessibility guidelines. Very often we find that existing tools may not be accurate measures when used with underserved populations. A close examination of how the tool works for individuals with specific disabilities or other underrepresented populations will increase the likelihood of obtaining useful information. Many individuals with disabilities have alternative methods of accessing information, utilizing assistive technologies such as screen readers or voice activation systems. Our survey instruments, measurement tools, and reporting mechanisms all need to be designed with this in mind.

Resources: Very little information in the evaluation literature exists specific to evaluating disability-based programs. Here are three disability related resources.

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

· ·

Hello, I am Gordon Bonham, owner of Bonham Research, Cranberry Township, Pennsylvania. Measuring the quality of life of people with intellectual and other developmental disabilities (ID/DD) presents a number of challenges.

Lessons Learned: Although it is generally assumed that self response is better than proxy response when people have the ability to respond for themselves, how many people with ID/DD can respond for themselves and who decides? The Maryland Ask Me! Survey has eight years of experience collecting quality of life information for about 1,300 individuals each year sampled from the roles of the state disability administration. The survey follows the principles of participatory action research further than other published studies on outcomes for this vulnerable population by employing only interviewers who are part of the population with ID/DD.

About 30 interviewers with ID/DD are employed each year to interview their peers, and they have worked for an average of 3.2 years. The peer interviewers are more likely to be young and female and have less severe intellectual disabilities and communication impairments, but more likely to have cerebral palsy than the peers they interview. However, they tend to have the same experiences with habilitation, employment, residential and other support services as those they interview. They work in teams of two that allows non-readers and people unable to record answers to interview. The teams make the determination if a person has the ability to respond for him or herself, and find that three-fourths can, including one-fifth of those classified with profound intellectual disabilities. The peer interviewers also interview the proxies for those unable to respond for themselves, generally impressing families and staff with their abilities.

The 90 peer interviewers who have worked over the eight years of the statewide evaluation have contributed greatly to the quality of data used to guide state policy, enhance agency services, and inform consumers making choices for where to get services. In addition, they have personally benefitted from participation in the evaluation. A survey shows that their employment in research increased peer interviewers’ self-confidence, improved communications skills and created openness to new opportunities. Employment as interviewers provided one-fourth of them with their first paid community job experience and helped one-fifth to subsequently step into a better job or pursue further education or training. Research employment also helped one-fourth to move into more independent living, expand friendships, increase participation in clubs and groups, and increase advocacy.

Rad Resource: ASK ME! Survey http://www.thearcmd.org/programs/ask_me.html

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

· ·

Greetings, I am June Gothberg, the State Technical Assistance Coordinator for the National Secondary Transition Technical Assistance Center (NSTTAC). I am also co-chair of AEA’s Disabilities and Other Vulnerable Populations TIG (DOVP). As we worked together on this week’s AEA365 blog, we wanted bring you practical resources, hints, and tricks for including people with disabilities and other vulnerable populations in your evaluations. DOVP is often approached with simple questions of “how to” in this area. My co-chair, Jennifer will be presenting an AEA Coffee Break Webinar on Thursday on how to include Universal Design in your evaluations and we will be also giving a full hands-on workshop at the AEA Conference in San Antonio with another of our colleagues. We would love to have you there!

In the meantime, I would encourage you to get to know the seven basic principals of Universal Design:

  1. Equitable use
  2. Flexibility in use
  3. Simple and intuitive use
  4. Perceptible information
  5. Tolerance for error
  6. Low physical effort
  7. Size and space for approach and use

Hot Tips: Today, I would like to suggest tips and resources from Principal 3: Simple and intuitive use. In other words keep it simple.

  • First and foremost, know your participants and prepare. If you know you have people involved who have hearing, vision, ambulatory, sensory challenges, make sure to accommodate ahead of time. Prepare materials and activities with them in mind.
  • Remember you may enjoy fancy fonts, backgrounds, and graphics, but the best tools are simple black and white. Besides people with low vision, 8% (10.5 million) of males in the United States are color blind making it difficult to read multicolor materials.
  • If are looking to include people with low reading abilities either create the evaluation materials with them in mind or create an alternative assessment at a lower reading level. A simple trick is to use the option provided in your word processing program to rate the ease of readability (MS Word uses the Flesch-Kincaid grade level) and keep your evaluation tool below a 4th to 6th grade reading level to include more diverse participation.
  • If you are looking to include people in a survey who are either non-verbal or low readers by all means create a survey that uses pictorial responses either in the form of smiles to frowns or actual pictures of activities.
  • Offer materials, handouts, and presentations in a variety of formats: large print or Braille if needed.
  • Be prepared to include a reader or interpreter.

Rad resources – Websites:

Rad resources – Free picture resources:

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

· ·

My name is Jennifer Sullivan Sulewski and I am a Research Associate at the Institute for Community Inclusion (ICI) at the University of Massachusetts Boston. At ICI I have worked on a variety of research and evaluation projects related to services and supports for people with disabilities. I currently am most involved with Work Without Limits, a public-private partnership funded by the Massachusetts Medicaid Infrastructure and Comprehensive Employment Opportunities (MI-CEO) grant to strengthen the Massachusetts workforce and advance work opportunities for youth and adults with disabilities in Massachusetts.

Universal Design refers to “the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (http://www.design.ncsu.edu/cud/about_ud/udprincipleshtmlformat.html). The principles of universal design can be applied to evaluation to ensure that all relevant populations are included at every stage of the work, from project design to sharing of findings.

Hot Tip: Universal design is helpful to think about even if your evaluation is not specifically focused on disability programs or issues. You will likely encounter people with disabilities or members of other vulnerable populations whatever your focus, so it is good to be prepared. Moreover, good universal design works better for everyone, even those without disabilities or other barriers. Think of bar patrons watching TV with closed captioning on, or people with strollers or rolling suitcases using elevators and curb cuts.

Hot Tip: One key aspect of applying universal design to evaluation work is to think about all the different ways people communicate or access information. For example, if you are conducting a survey on-line, you might offer the option to do it on paper or over the phone if respondents prefer not to respond by computer. Or if you are doing interviews by phone you may find that some respondents prefer to speak in person or respond by e-mail.

Rad Resource: For more tips, join the AEA Coffee Break Webinar on this topic on Thursday, August 5 at 2PM – click here to learn more and sign up. The Disability and Other Vulnerable Populations TIG is also hosting multiple sessions related to Universal Design, including a skill building workshop and a roundtable, at the upcoming American Evaluation Association Annual Conference in San Antonio the first week in November, so look for us there!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

· · ·

Archives

To top