DUP Week: Accessible Evaluation Techniques with June Gothberg

Greetings, I am June Gothberg, Ph.D. from Western Michigan University, Chair of the Disabilities and Underrepresented Populations TIG and co-author of the Universal Design for Evaluation Checklist (4th ed.).   Historically, our TIG has been a ‘working’ TIG, working collaboratively with AEA and the field to build capacity for accessible and inclusive evaluation.  Several terms tend to describe our philosophy – inclusive, accessible, perceptible, voice, empowered, equitable, representative, to name a few.  As we end our week, I’d like to share major themes that have emerged over my three terms in TIG leadership.

Lessons Learned

  • Representation in evaluation should mirror representation in the program. Oftentimes, this can be overlooked in evaluation reports.  This is an example from a community housing evaluation.  The data overrepresented some groups and underrepresented others.

 HUD Participant Data Comparison

  • Avoid using TDMs.
    • T = tokenism or giving participants a voice in evaluation efforts but little to no choice about the subject, style of communication, or any say in the organization.
    • D = decoration or asking participants to take part in evaluation efforts with little to no explanation of the reason for their involvement or its use.
    • M = manipulation or manipulating participants to participate in evaluation efforts. One example was presented in 2010 where food stamp recipients were required to answer surveys or they were ineligible to continue receiving assistance.  The surveys included identifying information.
  • Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.
    • Embed the principals of Universal Design whenever and wherever possible.
    • Utilize trauma-informed practice.
  • Increase authentic participation, voice, recommendations, and decision-making by engaginge all types and levels of stakeholders in evaluation planning efforts. The IDEA Partnership depth of engagement framework for program planning and evaluation has been adopted in state government planning efforts across the United States.

 IDEA Partnership Leading by Convening Framework

  • Disaggregating data helps uncover and eliminate inequities. This example is data from Detroit Public Schools (DPS).  DPS is in the news often and cited as having dismal outcomes.  If we were to compare state data with DPS, does it really look dismal?2015-16 Graduation and Dropout Rates


Disaggregating by one level would uncover some inequities, but disaggregating by two levels shows areas that can and should be addressed.2015-16_Grad_DO_rate_DTW_M_F



We hope you’ve enjoyed this week of aea365 hosted by the DUP TIG.  We’d love to have you join us at AEA 2017 and throughout the year.

The American Evaluation Association is hosting the Disabilities and Underrepresented Populations TIG (DUP) Week. The contributions all week are focused on engaging DUP in your evaluation efforts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

3 thoughts on “DUP Week: Accessible Evaluation Techniques with June Gothberg”

  1. Laurel Charnetski

    Dear Dr. Gothberg,
    Thank you so much for your contribution and very timely as I am evaluating an elementary school program in which 50% of the participants are disabled. If I do not adequately ensure every voice is heard, I risk alienating 50% of my stakeholders. Trough reading your post I realize that I must create access and time to ensure all stakeholders are represented when doing my evaluation.I really appreciated your statement that we need to “increase authentic participation, voice, recommendations, and decision-making by engaging all types and levels of stakeholders in evaluation planning efforts.” A few participants in the program have expressive language difficulties and I would appreciate any suggestions you might have in regards to collecting meaningful input.There are many facets to consider when developing an evaluation that allows equitable access and representation and I appreciate that you and the DUP are focused in this area and the other contributors on A365 during the week have been terrific.
    All my best, Laurel Charnetski

  2. Alia Hamadani

    Hello Ms. Gothberg,

    My name is Alia Hamadani and I am pursing my graduate studies in the Professional Masters of Education program at Queen’s University in Ontario, Canada. I am currently enrolled in the Program Inquiry and Evaluation course and was asked to connect with the evaluation community. As I am new to the evaluation world I am intrigued with all it encompasses and am eager to learn from professionals like you. Your article really resonated with me because of the keen interests I have in creating a PE that is equitable and in favour of all its stakeholders. I find this a little challenging, as there are many factors to consider when doing so. I appreciate the “lessons learned” you have mentioned as they have shed some light onto how this can be done and what to consider. However some questions that come to mind are “how does one consider all stakeholders if their interests and needs are different? If evaluation is required to focus on intended users or primary users and to create goals and expected outcomes collaboratively based on these, can it be done effectively?

    You mention “Increase authentic participation, voice, recommendations, and decision-making by engaging all types and levels of stakeholders in evaluation planning efforts.” How do we encourage stakeholders to engage in evaluation? The “Doing the Work Together Rubric” was very thought provoking with the different levels of interaction and provided me with a valuable resource to refer to when creating my own PE. How do we determine which stakeholders are involved at each level?

    Your insight is greatly appreciated.

  3. Hey Jane Gothberg, I am currently taking my Masters’ of Education through Queens University, and my current assignment, is to find an article that inspires me and write a reply to the author.

    I was inspired by your article because of the importance you stress on ‘voice’ (this theme really stuck out to the most in your article) in terms of making sure that all voices are heard and that results from an evaluation are presented to all stakeholders.

    I have two questions/comments to make.

    1) Hearing all voices and including all stakeholders is important, but what do you do when time and resources are limited? How do you ensure that you are including all stakeholders ‘voices?

    2) I was wondering as well, how do you communicate to those who from different cultures, who might speak a different language or because of cultural nuances might not be as open to communicating criticisms?

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.