DOVP Week: Mary Moriarty on Planning and Implementing Disability-based Evaluations

I’m Mary Moriarty, independent consultant and evaluator with Picker Engineering Program at Smith College. For 10 years I have specialized in evaluation of programs that serve underrepresented populations, particularly in science, technology, engineering, and mathematics (STEM). I previously directed several programs focused on increasing representation of individuals with disabilities in STEM.

I now realize the importance of ensuring cultural relevancy for effective project evaluation. Nowhere is this more critical than disability-based evaluations where contextual factors impact all phases of the evaluation. Here are some tips helpful in planning and implementing disability-based evaluations.

Hot Tip – Understand the Population: One of the most critical factors is determining impact on the populations being examined. However, in disability programs there can be significant disparities in definitions and classification systems. Some projects use definitions provided by the Americans with Disabilities Act others use internal or funding agency definitions. Comparing data becomes confusing or difficult, particularly when working with multiple agencies or programs. As evaluators we need to be aware of these differences so we can provide clarity and direction to the evaluation process.

Hot Tip – Understand the Impact of Differences: No two individuals with disabilities are alike; therefore evaluators need to understand the range and types of disabilities. Differences may present challenges on many fronts. First, developing comparison measures can be difficult when there are significant differences between individuals within the population. For example, the experience of an individual who uses a wheel chair may be different than that of an individual with a learning disability. Second, many individuals with disabilities have experienced some level of discrimination and may be reluctant to disclose sensitive information. There may be issues around confidentially or disclosure that could impact evaluation results. Being sensitive to these issues, establishing rapport, and utilizing a wide range of qualitative and quantitative measures will help to ensure the collection of accurate and useful data.

Hot Tip -Design Tools, Assessment Measures, and Surveys that are Universally Accessible: Third, we need to ensure that all evaluation methods and measures meet accessibility guidelines. Very often we find that existing tools may not be accurate measures when used with underserved populations. A close examination of how the tool works for individuals with specific disabilities or other underrepresented populations will increase the likelihood of obtaining useful information. Many individuals with disabilities have alternative methods of accessing information, utilizing assistive technologies such as screen readers or voice activation systems. Our survey instruments, measurement tools, and reporting mechanisms all need to be designed with this in mind.

Resources: Very little information in the evaluation literature exists specific to evaluating disability-based programs. Here are three disability related resources.

• Association on Higher Education and Disability
• Center for Applied Special Technology
• National Science Foundation/Research in Disabilities Education

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.