I’m Mary Moriarty, independent consultant and evaluator with Picker Engineering Program at Smith College. For 10 years I have specialized in evaluation of programs that serve underrepresented populations, particularly in science, technology, engineering, and mathematics (STEM). I previously directed several programs focused on increasing representation of individuals with disabilities in STEM.

I now realize the importance of ensuring cultural relevancy for effective project evaluation. Nowhere is this more critical than disability-based evaluations where contextual factors impact all phases of the evaluation. Here are some tips helpful in planning and implementing disability-based evaluations.

Hot Tip – Understand the Population: One of the most critical factors is determining impact on the populations being examined. However, in disability programs there can be significant disparities in definitions and classification systems. Some projects use definitions provided by the Americans with Disabilities Act others use internal or funding agency definitions. Comparing data becomes confusing or difficult, particularly when working with multiple agencies or programs. As evaluators we need to be aware of these differences so we can provide clarity and direction to the evaluation process.

Hot Tip – Understand the Impact of Differences: No two individuals with disabilities are alike; therefore evaluators need to understand the range and types of disabilities. Differences may present challenges on many fronts. First, developing comparison measures can be difficult when there are significant differences between individuals within the population. For example, the experience of an individual who uses a wheel chair may be different than that of an individual with a learning disability. Second, many individuals with disabilities have experienced some level of discrimination and may be reluctant to disclose sensitive information. There may be issues around confidentially or disclosure that could impact evaluation results. Being sensitive to these issues, establishing rapport, and utilizing a wide range of qualitative and quantitative measures will help to ensure the collection of accurate and useful data.

Hot Tip -Design Tools, Assessment Measures, and Surveys that are Universally Accessible: Third, we need to ensure that all evaluation methods and measures meet accessibility guidelines. Very often we find that existing tools may not be accurate measures when used with underserved populations. A close examination of how the tool works for individuals with specific disabilities or other underrepresented populations will increase the likelihood of obtaining useful information. Many individuals with disabilities have alternative methods of accessing information, utilizing assistive technologies such as screen readers or voice activation systems. Our survey instruments, measurement tools, and reporting mechanisms all need to be designed with this in mind.

Resources: Very little information in the evaluation literature exists specific to evaluating disability-based programs. Here are three disability related resources.

• Association on Higher Education and Disability
• Center for Applied Special Technology
• National Science Foundation/Research in Disabilities Education

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

context, Disabilities, Evaluation planning

Hello, I am Gordon Bonham, owner of Bonham Research, Cranberry Township, Pennsylvania. Measuring the quality of life of people with intellectual and other developmental disabilities (ID/DD) presents a number of challenges.

Lessons Learned: Although it is generally assumed that self response is better than proxy response when people have the ability to respond for themselves, how many people with ID/DD can respond for themselves and who decides? The Maryland Ask Me! Survey has eight years of experience collecting quality of life information for about 1,300 individuals each year sampled from the roles of the state disability administration. The survey follows the principles of participatory action research further than other published studies on outcomes for this vulnerable population by employing only interviewers who are part of the population with ID/DD.

About 30 interviewers with ID/DD are employed each year to interview their peers, and they have worked for an average of 3.2 years. The peer interviewers are more likely to be young and female and have less severe intellectual disabilities and communication impairments, but more likely to have cerebral palsy than the peers they interview. However, they tend to have the same experiences with habilitation, employment, residential and other support services as those they interview. They work in teams of two that allows non-readers and people unable to record answers to interview. The teams make the determination if a person has the ability to respond for him or herself, and find that three-fourths can, including one-fifth of those classified with profound intellectual disabilities. The peer interviewers also interview the proxies for those unable to respond for themselves, generally impressing families and staff with their abilities.

The 90 peer interviewers who have worked over the eight years of the statewide evaluation have contributed greatly to the quality of data used to guide state policy, enhance agency services, and inform consumers making choices for where to get services. In addition, they have personally benefitted from participation in the evaluation. A survey shows that their employment in research increased peer interviewers’ self-confidence, improved communications skills and created openness to new opportunities. Employment as interviewers provided one-fourth of them with their first paid community job experience and helped one-fifth to subsequently step into a better job or pursue further education or training. Research employment also helped one-fourth to move into more independent living, expand friendships, increase participation in clubs and groups, and increase advocacy.

Rad Resource: ASK ME! Survey http://www.thearcmd.org/programs/ask_me.html

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

Disabilities, interviewing, participatory action research

Greetings, I am June Gothberg, the State Technical Assistance Coordinator for the National Secondary Transition Technical Assistance Center (NSTTAC). I am also co-chair of AEA’s Disabilities and Other Vulnerable Populations TIG (DOVP). As we worked together on this week’s AEA365 blog, we wanted bring you practical resources, hints, and tricks for including people with disabilities and other vulnerable populations in your evaluations. DOVP is often approached with simple questions of “how to” in this area. My co-chair, Jennifer will be presenting an AEA Coffee Break Webinar on Thursday on how to include Universal Design in your evaluations and we will be also giving a full hands-on workshop at the AEA Conference in San Antonio with another of our colleagues. We would love to have you there!

In the meantime, I would encourage you to get to know the seven basic principals of Universal Design:

1. Equitable use
2. Flexibility in use
3. Simple and intuitive use
4. Perceptible information
5. Tolerance for error
6. Low physical effort
7. Size and space for approach and use

Hot Tips: Today, I would like to suggest tips and resources from Principal 3: Simple and intuitive use. In other words keep it simple.

• First and foremost, know your participants and prepare. If you know you have people involved who have hearing, vision, ambulatory, sensory challenges, make sure to accommodate ahead of time. Prepare materials and activities with them in mind.
• Remember you may enjoy fancy fonts, backgrounds, and graphics, but the best tools are simple black and white. Besides people with low vision, 8% (10.5 million) of males in the United States are color blind making it difficult to read multicolor materials.
• If are looking to include people with low reading abilities either create the evaluation materials with them in mind or create an alternative assessment at a lower reading level. A simple trick is to use the option provided in your word processing program to rate the ease of readability (MS Word uses the Flesch-Kincaid grade level) and keep your evaluation tool below a 4^th to 6^th grade reading level to include more diverse participation.
• If you are looking to include people in a survey who are either non-verbal or low readers by all means create a survey that uses pictorial responses either in the form of smiles to frowns or actual pictures of activities.
• Offer materials, handouts, and presentations in a variety of formats: large print or Braille if needed.
• Be prepared to include a reader or interpreter.

Rad resources – Websites:

• Universal Design Principles: http://www.design.ncsu.edu/cud/about_ud/udprinciplestext.htm
• Web Content Accessibility Guidelines: http://www.w3.org/WAI/intro/wcag.php
• Presentation Accessibility: http://www.w3.org/WAI/training/accessible

Rad resources – Free picture resources:

• Smiles for surveys: http://www.nsttac.org/products_and_resources/EvaluationToolkit/
• Do to Learn picture resources http://www.dotolearn.com/

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Universal Design

My name is Jennifer Sullivan Sulewski and I am a Research Associate at the Institute for Community Inclusion (ICI) at the University of Massachusetts Boston. At ICI I have worked on a variety of research and evaluation projects related to services and supports for people with disabilities. I currently am most involved with Work Without Limits, a public-private partnership funded by the Massachusetts Medicaid Infrastructure and Comprehensive Employment Opportunities (MI-CEO) grant to strengthen the Massachusetts workforce and advance work opportunities for youth and adults with disabilities in Massachusetts.

Universal Design refers to “the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (http://www.design.ncsu.edu/cud/about_ud/udprincipleshtmlformat.html). The principles of universal design can be applied to evaluation to ensure that all relevant populations are included at every stage of the work, from project design to sharing of findings.

Hot Tip: Universal design is helpful to think about even if your evaluation is not specifically focused on disability programs or issues. You will likely encounter people with disabilities or members of other vulnerable populations whatever your focus, so it is good to be prepared. Moreover, good universal design works better for everyone, even those without disabilities or other barriers. Think of bar patrons watching TV with closed captioning on, or people with strollers or rolling suitcases using elevators and curb cuts.

Hot Tip: One key aspect of applying universal design to evaluation work is to think about all the different ways people communicate or access information. For example, if you are conducting a survey on-line, you might offer the option to do it on paper or over the phone if respondents prefer not to respond by computer. Or if you are doing interviews by phone you may find that some respondents prefer to speak in person or respond by e-mail.

Rad Resource: For more tips, join the AEA Coffee Break Webinar on this topic on Thursday, August 5 at 2PM – click here to learn more and sign up. The Disability and Other Vulnerable Populations TIG is also hosting multiple sessions related to Universal Design, including a skill building workshop and a roundtable, at the upcoming American Evaluation Association Annual Conference in San Antonio the first week in November, so look for us there!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Inclusive, Universal Design