I’m Mary Moriarty, independent consultant and evaluator with Picker Engineering Program at Smith College. For 10 years I have specialized in evaluation of programs that serve underrepresented populations, particularly in science, technology, engineering, and mathematics (STEM). I previously directed several programs focused on increasing representation of individuals with disabilities in STEM.

I now realize the importance of ensuring cultural relevancy for effective project evaluation. Nowhere is this more critical than disability-based evaluations where contextual factors impact all phases of the evaluation. Here are some tips helpful in planning and implementing disability-based evaluations.

Hot Tip – Understand the Population: One of the most critical factors is determining impact on the populations being examined. However, in disability programs there can be significant disparities in definitions and classification systems. Some projects use definitions provided by the Americans with Disabilities Act others use internal or funding agency definitions. Comparing data becomes confusing or difficult, particularly when working with multiple agencies or programs. As evaluators we need to be aware of these differences so we can provide clarity and direction to the evaluation process.

Hot Tip – Understand the Impact of Differences: No two individuals with disabilities are alike; therefore evaluators need to understand the range and types of disabilities. Differences may present challenges on many fronts. First, developing comparison measures can be difficult when there are significant differences between individuals within the population. For example, the experience of an individual who uses a wheel chair may be different than that of an individual with a learning disability. Second, many individuals with disabilities have experienced some level of discrimination and may be reluctant to disclose sensitive information. There may be issues around confidentially or disclosure that could impact evaluation results. Being sensitive to these issues, establishing rapport, and utilizing a wide range of qualitative and quantitative measures will help to ensure the collection of accurate and useful data.

Hot Tip -Design Tools, Assessment Measures, and Surveys that are Universally Accessible: Third, we need to ensure that all evaluation methods and measures meet accessibility guidelines. Very often we find that existing tools may not be accurate measures when used with underserved populations. A close examination of how the tool works for individuals with specific disabilities or other underrepresented populations will increase the likelihood of obtaining useful information. Many individuals with disabilities have alternative methods of accessing information, utilizing assistive technologies such as screen readers or voice activation systems. Our survey instruments, measurement tools, and reporting mechanisms all need to be designed with this in mind.

Resources: Very little information in the evaluation literature exists specific to evaluating disability-based programs. Here are three disability related resources.

• Association on Higher Education and Disability
• Center for Applied Special Technology
• National Science Foundation/Research in Disabilities Education

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

context, Disabilities, Evaluation planning

My name is Aimee Sickels and I am the Principal Evaluator and owner of Custom Evaluation Services, an independent evaluation consulting business. I am also currently working on a Ph.D. in Social Work at the University of South Carolina. My helpful hint relates to the use of logic models. I am an empowerment evaluator and believe that our role as evaluators is not only to evaluate the programs, but to build the evaluation capacity in the organizations we evaluate. So, how can the logic model help?

Hot Tip: The 2 minute logic model. I will often look over my client’s programming before I am going on a site visit or a monitoring meeting. I will select a single program item, it may be a single goal or a single activity, and I will bring this to my client as the 2 minute logic model. Before we get started with our planned meeting I present my single item to them, for example one of my client’s who serve vulnerable persons has a goal of actually increasing involvement of the client in the program design; having the client’s participate more in their own service design. I start this on the input column and walk the client through that single item across the output column and ultimately the outcome column. I then say spend the rest of today and the rest of this week working on this single item. Because it is a single item and a single line it is easy to follow and quick to see and understand. Put the 2 minute logic model write up at each staff person’s desk and ask them to consider it with each client they see. I can guarantee that they will see an improvement on that line of programming for that week! Try the 2 minute logic model today!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

empowerment, logic models

Hello, I am Gordon Bonham, owner of Bonham Research, Cranberry Township, Pennsylvania. Measuring the quality of life of people with intellectual and other developmental disabilities (ID/DD) presents a number of challenges.

Lessons Learned: Although it is generally assumed that self response is better than proxy response when people have the ability to respond for themselves, how many people with ID/DD can respond for themselves and who decides? The Maryland Ask Me! Survey has eight years of experience collecting quality of life information for about 1,300 individuals each year sampled from the roles of the state disability administration. The survey follows the principles of participatory action research further than other published studies on outcomes for this vulnerable population by employing only interviewers who are part of the population with ID/DD.

About 30 interviewers with ID/DD are employed each year to interview their peers, and they have worked for an average of 3.2 years. The peer interviewers are more likely to be young and female and have less severe intellectual disabilities and communication impairments, but more likely to have cerebral palsy than the peers they interview. However, they tend to have the same experiences with habilitation, employment, residential and other support services as those they interview. They work in teams of two that allows non-readers and people unable to record answers to interview. The teams make the determination if a person has the ability to respond for him or herself, and find that three-fourths can, including one-fifth of those classified with profound intellectual disabilities. The peer interviewers also interview the proxies for those unable to respond for themselves, generally impressing families and staff with their abilities.

The 90 peer interviewers who have worked over the eight years of the statewide evaluation have contributed greatly to the quality of data used to guide state policy, enhance agency services, and inform consumers making choices for where to get services. In addition, they have personally benefitted from participation in the evaluation. A survey shows that their employment in research increased peer interviewers’ self-confidence, improved communications skills and created openness to new opportunities. Employment as interviewers provided one-fourth of them with their first paid community job experience and helped one-fifth to subsequently step into a better job or pursue further education or training. Research employment also helped one-fourth to move into more independent living, expand friendships, increase participation in clubs and groups, and increase advocacy.

Rad Resource: ASK ME! Survey http://www.thearcmd.org/programs/ask_me.html

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

Disabilities, interviewing, participatory action research

My name is David Merves and I work for Evergreen Educational Consulting LLC (EEC), an evaluation consulting firm. A part of our practice involves evaluating federally funded, Special Education grants and programs that are intended to improve education and services for children and youth at risk of school failure or in need of special education (e.g., IDEA, Section 504).

The advent of online learning, distance learning and use of technology for the classroom has transformed how our clients work and the strategies and processes they employ. In order to perform our evaluative functions EEC is committed to staying abreast of technology issues/changes as they relate to Special Education. How can we evaluate that which we do not understand?

Hot Tip - iNACOL – International Association for K-12 Online Learning: iNACOL has excellent resources for collaboration, advocacy, and research to enhance quality K-12 online teaching and learning for all students.

Hot Tip – ISTE – International Society for Technology in Education : ISTE is an association for educators and education leaders engaged in improving teaching and learning by advancing the effective use of technology in Pre K-12 and teacher education.

Rad Resources: If you wish to dive deeper into technology waters here are three additional resources, which, go beyond Pre K-12 and bridge the transition to the working world:

1. Christopher Dede- Harvard Graduate School of Education He states, education needs to prepare today’s students for tomorrow’s jobs, which may not, currently, have been invented/created. He uses Web 2.0 as a jumping off point to discuss “thinking skills” for teachers and students utilizing tools, such as, wikis, blogs and discussion forums. He goes further and discusses creativity skills using video/photo sharing websites, mashups and online writer workshops.
2. Elliott Masie is an internationally recognized futurist, analyst, researcher and organizer on the critical topics of workforce learning, business collaboration and emerging technologies. He has a free e newsletter, as well, as fee-based services.
3. The New Division of Labor: How Computers Are Creating the Next Job Market (book), Frank Levy (Author), Richard J. Murnane (Author). Their premise is, if educational curriculums aren’t changed to reflect the market’s demand for sophisticated thinking and communication, students may graduate without the skills they need.

Google the above names for further materials and references.

These resources have enhanced our abilities as evaluators to work in the changing and increasingly technological-educational world. We as evaluators need to stay current with the methods and processes being adapted for use within Special Education. One example is the promising research being demonstrated in Second Life (a MUVE- Multi-user Virtual Environment), with students on the Autism Spectrum at the Center for BrainHealth at the University of Texas at Dallas.

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

special education, technology

Greetings, I am June Gothberg, the State Technical Assistance Coordinator for the National Secondary Transition Technical Assistance Center (NSTTAC). I am also co-chair of AEA’s Disabilities and Other Vulnerable Populations TIG (DOVP). As we worked together on this week’s AEA365 blog, we wanted bring you practical resources, hints, and tricks for including people with disabilities and other vulnerable populations in your evaluations. DOVP is often approached with simple questions of “how to” in this area. My co-chair, Jennifer will be presenting an AEA Coffee Break Webinar on Thursday on how to include Universal Design in your evaluations and we will be also giving a full hands-on workshop at the AEA Conference in San Antonio with another of our colleagues. We would love to have you there!

In the meantime, I would encourage you to get to know the seven basic principals of Universal Design:

1. Equitable use
2. Flexibility in use
3. Simple and intuitive use
4. Perceptible information
5. Tolerance for error
6. Low physical effort
7. Size and space for approach and use

Hot Tips: Today, I would like to suggest tips and resources from Principal 3: Simple and intuitive use. In other words keep it simple.

• First and foremost, know your participants and prepare. If you know you have people involved who have hearing, vision, ambulatory, sensory challenges, make sure to accommodate ahead of time. Prepare materials and activities with them in mind.
• Remember you may enjoy fancy fonts, backgrounds, and graphics, but the best tools are simple black and white. Besides people with low vision, 8% (10.5 million) of males in the United States are color blind making it difficult to read multicolor materials.
• If are looking to include people with low reading abilities either create the evaluation materials with them in mind or create an alternative assessment at a lower reading level. A simple trick is to use the option provided in your word processing program to rate the ease of readability (MS Word uses the Flesch-Kincaid grade level) and keep your evaluation tool below a 4^th to 6^th grade reading level to include more diverse participation.
• If you are looking to include people in a survey who are either non-verbal or low readers by all means create a survey that uses pictorial responses either in the form of smiles to frowns or actual pictures of activities.
• Offer materials, handouts, and presentations in a variety of formats: large print or Braille if needed.
• Be prepared to include a reader or interpreter.

Rad resources – Websites:

• Universal Design Principles: http://www.design.ncsu.edu/cud/about_ud/udprinciplestext.htm
• Web Content Accessibility Guidelines: http://www.w3.org/WAI/intro/wcag.php
• Presentation Accessibility: http://www.w3.org/WAI/training/accessible

Rad resources – Free picture resources:

• Smiles for surveys: http://www.nsttac.org/products_and_resources/EvaluationToolkit/
• Do to Learn picture resources http://www.dotolearn.com/

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Universal Design

My name is Jennifer Sullivan Sulewski and I am a Research Associate at the Institute for Community Inclusion (ICI) at the University of Massachusetts Boston. At ICI I have worked on a variety of research and evaluation projects related to services and supports for people with disabilities. I currently am most involved with Work Without Limits, a public-private partnership funded by the Massachusetts Medicaid Infrastructure and Comprehensive Employment Opportunities (MI-CEO) grant to strengthen the Massachusetts workforce and advance work opportunities for youth and adults with disabilities in Massachusetts.

Universal Design refers to “the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (http://www.design.ncsu.edu/cud/about_ud/udprincipleshtmlformat.html). The principles of universal design can be applied to evaluation to ensure that all relevant populations are included at every stage of the work, from project design to sharing of findings.

Hot Tip: Universal design is helpful to think about even if your evaluation is not specifically focused on disability programs or issues. You will likely encounter people with disabilities or members of other vulnerable populations whatever your focus, so it is good to be prepared. Moreover, good universal design works better for everyone, even those without disabilities or other barriers. Think of bar patrons watching TV with closed captioning on, or people with strollers or rolling suitcases using elevators and curb cuts.

Hot Tip: One key aspect of applying universal design to evaluation work is to think about all the different ways people communicate or access information. For example, if you are conducting a survey on-line, you might offer the option to do it on paper or over the phone if respondents prefer not to respond by computer. Or if you are doing interviews by phone you may find that some respondents prefer to speak in person or respond by e-mail.

Rad Resource: For more tips, join the AEA Coffee Break Webinar on this topic on Thursday, August 5 at 2PM – click here to learn more and sign up. The Disability and Other Vulnerable Populations TIG is also hosting multiple sessions related to Universal Design, including a skill building workshop and a roundtable, at the upcoming American Evaluation Association Annual Conference in San Antonio the first week in November, so look for us there!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Inclusive, Universal Design

My name is Gerri Spilka and I am the Executive Director of the OMG Center for Collaborative Learning – a consulting firm that assists and assesses change through evaluation, strategic research and planning, grants management and capacity building. Among our many projects, OMG serves as the lead management team on the Robert Wood Johnson Foundation Evaluation Fellowship Program and it is about that program that I will share information today.

The skill set needed to interpret effectively public health, social service, community and organizational social patterns depends on the authentic and diverse experiences within program evaluation teams. Programs and services that are often evaluated traditionally work with underserved and disadvantaged communities, which are themselves becoming more diverse. However, the program evaluation field is becoming less diverse due to a lack of training and placement opportunities for emerging professionals from graduate programs who come from underserved and disadvantaged communities. This creates a problem for the field where, more often than not, high-quality evaluations reflect a process that incorporates diverse perspectives. The RWJF Evaluation Fellowship Program works to ameliorate this disparity.

Resource: The RWJF Evaluation Fellowship Program seeks to extend the skills of participants from diverse backgrounds to include culturally responsive evaluation practices. The Fellows program supports two cohorts annually on parallel paths:

• The Emerging Professionals Program is aimed at those who have graduated from a master’s or doctoral degree program within the last three years and provides a one-year placement, extensive training opportunities, and a support stipend
• The Retooling Professionals Program is focused on mid- or senior-level nonprofit professionals who remain in their current position, but receive additional training to extend their skills to evaluation. Their organization receives a $5,000 financial award and the professional receives a travel stipend.

Both cohorts receive customized training, facilitated by some of today’s thought leaders in field, as well as mentorship, and practical field-based guidance.

Hot Tip: While the deadline from participating in the Retooling Professionals program has passed, the deadline for applying for the Emerging Professionals program is coming up on July 31, 2010. The Emerging Professionals application may be downloaded online from http://www.rwjf-evaluationfellows.org/emerging-fellows.

Resource: The RWJF Evaluation Fellowship Program just released a video entitled Evaluation With a Diversity Lens. This extended discourse among a range of professionals explores the value of diversity in the evaluation profession and may be downloaded for free.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, diversity

My name is Lija Greenseid. I am a Senior Evaluator, with Professional Data Analysts, Inc. in Minneapolis, MN. We conduct evaluations of stop-smoking programs. Smokers generally have lower education and literacy levels than the general population. Therefore, we want to make sure the materials we develop are understandable to smokers.

Rad Resource: Use a “readability calculator” to check the reading-level of your written materials. I have used this with program registration forms, survey instruments, consent statements, and other materials. Not surprisingly, the first drafts of my materials are often written at a level only grad students (and evaluators) can understand. With a critical eye and a few tweaks I can often rewrite my materials so that they are at an eighth-grade reading level, much more accessible to the people with whom I want to communicate.

A good Readability Calculator can be found here:

http://www.editcentral.com/gwt1/EditCentral.html

It provides you with both a reading ease score, and a number of different measures of the US school grade level of the text.

This blog posting is rated at a high-school reading level. Do you agree?

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

esl, Instruments, Literacy, Readability, reports, Smoking, surveys

My name is Rodney Hopson. I am a faculty member in the Department of Educational Foundations and Leadership in the School of Education at Duquesne University. For the last several years (with a host of great colleagues in AEA, with Torres Consulting Group and OMG Center for Collaborative Learning with the support of the National Science Foundation and the Robert Wood Johnson Foundation, I have been developing internship and fellowship programs to support graduate and post-graduate students of color and from traditionally underrepresented communities. Embedded in their practice is the understanding and development of Culturally Responsive Evaluation (CRE) approaches and practices in these same communities.

Rad Resource: Adapting the elements of evaluation practice into one that reflects a culturally responsive one, Karen Kirkhart, social work professor at Syracuse University, and I have presented at the AEA/CDC Summer Institute for the last few years. We use the Frierson, Hood, & Hughes chapter in the 2002 National Science Foundation User-Friendly Handbook for Project Evaluation to guide a deeper understanding of how to integrate CRE throughout the evaluation.

Hot Tip: Situate CRE within elements or a framework of evaluation. Whether you use the CDC Evaluation Steps, or another framework to describe steps in evaluation, the key is to embed CRE throughout. For instance, at each stage of evaluation, CRE should be present from the time we prepare for the evaluation to the dissemination and use of results. Below is the CRE framework (with appreciation for the support of Elizabeth Kahl, Syracuse University, who assisted us on graphic and technical design elements), adapted from Frierson, et.al, 2002:

Rad Resource: Learn more about fellowship and internship programs in culturally responsive evaluation through the RWJF Evaluation Fellowship website or AEA’s Graduate Education Diversity Internship website.

Rad Resource: Develop a better theoretical and practical explanation of CRE by reading about its origins, its aim, and its movement as reflected in chapters in any of the following books and special issues:

• The Role of Culture and Cultural Context in Evaluation (Hood, Hopson, and Frierson Eds., Information Age Publishing)
• The SAGE International Handbook of Educational Evaluation (Ryan and Cousins, Eds, SAGE.)
• Handbook of Social Research Ethics (Mertens and Ginsberg, Eds., SAGE)
• SAGE Handbook of Evaluation (Shaw, Greene, and Mark, Eds., SAGE.)
• The Handbook of Leadership Development Evaluation (Hannum, Martineau and Reinelt, Eds., Jossey-Bass)
• In Search of Cultural Competence in Evaluation (Thompson-Robinson, Hopson, Sen Gupta, Eds, Jossey-Bass)
• Responsive Evaluation (Greene and Abma, Eds., Jossey-Bass)

Want to hear more from Rodney Hopson? He is serving as this week’s Thought Leader on the AEA Thought Leader Discussion Series. Learn more at http://www.eval.org/thought_leaders.asp. This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, culture