AEA365 | A Tip-a-Day by and for Evaluators

TAG | Universal Design for Evaluation

Greetings, I am June Gothberg, Ph.D. from Western Michigan University, Chair of the Disabilities and Underrepresented Populations TIG and co-author of the Universal Design for Evaluation Checklist (4th ed.).   Historically, our TIG has been a ‘working’ TIG, working collaboratively with AEA and the field to build capacity for accessible and inclusive evaluation.  Several terms tend to describe our philosophy – inclusive, accessible, perceptible, voice, empowered, equitable, representative, to name a few.  As we end our week, I’d like to share major themes that have emerged over my three terms in TIG leadership.

Lessons Learned

  • Representation in evaluation should mirror representation in the program. Oftentimes, this can be overlooked in evaluation reports.  This is an example from a community housing evaluation.  The data overrepresented some groups and underrepresented others.

 HUD Participant Data Comparison

  • Avoid using TDMs.
    • T = tokenism or giving participants a voice in evaluation efforts but little to no choice about the subject, style of communication, or any say in the organization.
    • D = decoration or asking participants to take part in evaluation efforts with little to no explanation of the reason for their involvement or its use.
    • M = manipulation or manipulating participants to participate in evaluation efforts. One example was presented in 2010 where food stamp recipients were required to answer surveys or they were ineligible to continue receiving assistance.  The surveys included identifying information.
  • Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.
    • Embed the principals of Universal Design whenever and wherever possible.
    • Utilize trauma-informed practice.
  • Increase authentic participation, voice, recommendations, and decision-making by engaginge all types and levels of stakeholders in evaluation planning efforts. The IDEA Partnership depth of engagement framework for program planning and evaluation has been adopted in state government planning efforts across the United States.

 IDEA Partnership Leading by Convening Framework

  • Disaggregating data helps uncover and eliminate inequities. This example is data from Detroit Public Schools (DPS).  DPS is in the news often and cited as having dismal outcomes.  If we were to compare state data with DPS, does it really look dismal?2015-16 Graduation and Dropout Rates

 

Disaggregating by one level would uncover some inequities, but disaggregating by two levels shows areas that can and should be addressed.2015-16_Grad_DO_rate_DTW_M_F

 

 

We hope you’ve enjoyed this week of aea365 hosted by the DUP TIG.  We’d love to have you join us at AEA 2017 and throughout the year.

The American Evaluation Association is hosting the Disabilities and Underrepresented Populations TIG (DUP) Week. The contributions all week are focused on engaging DUP in your evaluation efforts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

I’m Pat Campbell, president of Campbell-Kibler Associates, Inc.  Under NSF funding, Eric Jolly, president of the Science Museum of  Minnesota and I, with the help of a lot of friends, have been generating research-based tips, such as those below, to improve the accuracy of data collection, the quality of the analysis and the appropriateness of the data collected over diverse populations.

Hot Tips:

  • Ask for demographic information ONLY at the end of measures. There may be exceptions in cases for people with disabilities who will need accommodations in order to complete the measures.
  • Have participants define their own race/ethnicity and disability status rather than having the identification done by data collectors or project/program staff.  If a standard set of categories for race/ethnicity and/or disability is used, also, in an open-ended question, ask participants to indicate their own race/ethnicity and disability status.
  • Have members of the target population review affective and psychosocial measures for clarity. Ask them what concepts they think are being measured. If what is being measured is obvious and there are sex, race, or disability stereotypes associated with the concepts, consider using a less obvious measure if an equally valid measure is available.
  • Be aware that there can be heterogeneity within subgroups. For example, while people who are visually impaired, hearing impaired, and learning disabled are all classified as having disabilities, the differences among them are very large and it might be appropriate to disaggregate by different categories of disability.
  • When race/ethnicity, gender, or disability status is used as an independent variable, specify the reason for its use and include the reason in documentation of the results.

Lessons Learned:

  • All populations are diverse:  The diversity may be in terms of race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status….  It may be visible or invisible. Most likely in every group there is a multiplicity of diversities.  High quality evaluations need to pay attention to the diversity of all populations being served.
  • Each individual is diverse.  As individuals, we have many demographic characteristics including our race, gender, ethnicity, age, geographic location, education, income, disability status, veteran status….  Rather than focusing on only one demographic category, high quality evaluations need to determine which categories are integral to the evaluation and focus on them.

Rad Resources:

  • Universal Design for Evaluation Checklist, 4th Edition.  The title, says it all.  Jennifer  Sullivan-Sulewski, & June Gothberg have developed a short planning tool that helps evaluators include people of all ages and all abilities in evaluations.
  • As soon as it goes live, we hope our website, Beyond Rigor will be another rad resource.  Let me know (Campbell@campbell-kibler.com) if you would like to be notified when that happens.

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · ·

Greeting, I am June Gothberg, Senior Researcher at Western Michigan University. You may also recognize me as the intern Lead Curator for aea365. My AEA internship is ending this month and I will be passing the torch to another. Look for forthcoming details from Susan Kistler.

My colleague, Jennifer Sullivan Sulewski and I co-chair the Disabilities and Other Vulnerable Populations TIG. We also co-authored the Universal Design for Evaluation (UDE) Checklist to assist evaluators to include individuals authentically in the evaluation process, people of all ages and abilities. The checklist is a tool for program evaluators who design, develop, implement, and disseminate evaluations. Today’s post focuses on numbers two and four of the checklist.

Lessons Learned:

  • Most evaluators intend to include everyone in the evaluation process.
  • In real life design and implementation, many evaluators fail to anticipate the needs of the diverse individuals important to the evaluation.

Hot Tips:

  • Design Evaluation using UDE Principle 2: Flexibility in Use. Evaluations that demonstrate flexibility in use accommodates a wide range of individual preferences and abilities. The evaluation plan will show evidence of preparation to:
    • Communicate with participants of diverse abilities, communication styles, and cultural backgrounds. (e.g., second language interpreters, sign language interpreters, readers, large text, and Braille)
    • Address individual needs.
    • Provide alternate data collection tools based on communication preferences and needs. (e.g., written, oral, using smart technology, observation)
    • Include extra time for participants with slower processing or language barriers.
    • Include extra time to observe cultural practices.
  • Design Evaluation using UDE Principle 4: Perceptible Information. The evaluation design that provides perceptible information will communicate necessary information effectively to the user, regardless of ambient conditions or the user’s sensory abilities.
    • Sensory issues are addressed. (lower lighting, no flickering florescent lights, minimal noise, seating away from doors and windows, quiet ‘fidget’ toys – think stress ball)
    • Multiple media options are used to present information.
    • All printed publications are available immediately or in a timely manner in alternate formats2.
    • A statement is included in all materials about procedures for requesting accommodations or assistance.
    • Online materials adhere to web accessibility standards.

Rad Resources:

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.e 

· ·

My name is John Kramer and I am currently a Research Fellow at the Institute for Community Inclusion, University of Massachusetts Boston. My work focuses on research and evaluation of employment outcomes of people with disabilities, participatory research, and aging issues for families of people with disabilities. I am a new member of the American Evaluation Association.

Tip:

  • Universal Design Principle 3 is “simple and intuitive”. Incorporating clear, simple language in writing while also providing concrete, every day examples improves access in two ways:
  1. it clarifies your intention as a writer and helps you focus on the basic idea you are trying to convey
  2. it allows for more stakeholder access and participation.

Hot tips:

  • Use plain language. This means substituting simpler words for more complex ones. It also means writing sentences that are free of excessive subordination. Also, try to avoid unnecessary modifiers like “really, totally, very, only, quite,” which may interfere with clarity.
  • Use concrete, accessible examples including images when helpful. Try to think of examples to illustrate your writing that are easy to picture and relate to. Using images is a good approach as well when appropriate.
  • Use clear, parallel examples in your writing. For instance, if you frame an example as noun, verb, recipient noun, then make sure all your examples use the same order of presentation.

Rad Resources

There are many good resources for how to incorporate plain language and images into your work. A few especially helpful ones around the web are:

  • Plainlanguage.gov -A website by the United States Federal government that gives some useful strategies and examples in using plain language.
  • Grammar Girl -A website that provides some basic tips and tricks to clarify your writing. Not for cognitive access per se, but elements can be useful in UD.
  • Picture Planner – A website that illustrates an example of how pictures can be used to facilitate cognitive access.
  • Creative Commons -Here you can find free pictures that you can use, often with attribution, to illustrate your work and writing.

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · · · ·

My name is Jennifer Sullivan Sulewski, Research Associate at the Institute for Community Inclusion, University of Massachusetts Boston. Most of my research and evaluation work has focused on improving employment and postsecondary education outcomes for people with disabilities. I am co-chair of AEA’s Disabilities and Other Vulnerable Populations (DOVP) TIG and co-author of the Universal Design for Evaluation Checklist. I have also had the pleasure of serving as curator for the DOVP Week.

There are two major Universal Design schools: one is broadly applied and the other specifically to curriculum and learning. Both can help inform evaluators and ensure accessibility for all evaluation participants. Each of this week’s posts focuses in on the concepts of Universal Design (UD)or Universal Design for Learning (UDL).

Hot Tip:

  1. Equitable Use
  2. Flexibility in Use
  3. Simple and Intuitive Use
  4. Perceptible Information
  5. Tolerance for Error
  6. Low Physical Effort
  7. Size and Space for Approach and Use
  1. Multiple Means of Representation
  2. Multiple Means of Action and Expression
  3. Multiple Means of Engagement

Lesson Learned:

  • Evaluation recruitment materials and informed consent must be accessible for authentic and ethical participation. In our Universal Design for Evaluation checklist, we demonstrate the importance of Principle 1: equitable use, particularly as it applies to the informed consent process. In my work with people with intellectual/developmental disabilities, I’ve learned how important it is to create recruitment and informed consent materials that are designed to be used and understood by all. For example, in an early project I had separate consent forms for different aspects of the project, and the need to sign multiple forms was confusing for participants. I learned to explain all the expectations and rights of participants in one simple form instead.  
  • Here is an example of such a consent form:

Universally Designed Informed Consent

  • It is also essential when working with this population to understand whether the participants are under someone else’s guardianship; if they are, consent must be obtained from the guardian and assent from the individual.

In the posts to follow, John Kramer discusses how to apply UD Principle 3 to increase access and stakeholder participation. David Bernstein describes applying Principles 3 and 7 to an evaluation involving Deaf-Blind program participants. June Gothberg demonstrates Principles 2 and 4 on flexible and perceptible information. Bob Hughes provides tips on evaluating UDL projects and Don Glass explores the use of UDL to guide the design and evaluation of curriculum, programs, and materials.

Rad Resource:

  • Looking for ideas on how to make your project more accessible to people of all backgrounds and abilities? The Universal Design for Evaluation Checklist is a resource for applying the seven principles of Universal Design to evaluation.

The American Evaluation Association is celebrating the Disabilities and Other Vulnerable Populations TIG (DOVP) Week. The contributions all week come from DOVP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

· · · ·

Archives

To top