DUP Week: Accessible Evaluation Techniques with June Gothberg

Greetings, I am June Gothberg, Ph.D. from Western Michigan University, Chair of the Disabilities and Underrepresented Populations TIG and co-author of the Universal Design for Evaluation Checklist (4th ed.).   Historically, our TIG has been a ‘working’ TIG, working collaboratively with AEA and the field to build capacity for accessible and inclusive evaluation.  Several terms tend to describe our philosophy – inclusive, accessible, perceptible, voice, empowered, equitable, representative, to name a few.  As we end our week, I’d like to share major themes that have emerged over my three terms in TIG leadership.

Lessons Learned

  • Representation in evaluation should mirror representation in the program. Oftentimes, this can be overlooked in evaluation reports.  This is an example from a community housing evaluation.  The data overrepresented some groups and underrepresented others.

 HUD Participant Data Comparison

  • Avoid using TDMs.
    • T = tokenism or giving participants a voice in evaluation efforts but little to no choice about the subject, style of communication, or any say in the organization.
    • D = decoration or asking participants to take part in evaluation efforts with little to no explanation of the reason for their involvement or its use.
    • M = manipulation or manipulating participants to participate in evaluation efforts. One example was presented in 2010 where food stamp recipients were required to answer surveys or they were ineligible to continue receiving assistance.  The surveys included identifying information.
  • Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.
    • Embed the principals of Universal Design whenever and wherever possible.
    • Utilize trauma-informed practice.
  • Increase authentic participation, voice, recommendations, and decision-making by engaginge all types and levels of stakeholders in evaluation planning efforts. The IDEA Partnership depth of engagement framework for program planning and evaluation has been adopted in state government planning efforts across the United States.

 IDEA Partnership Leading by Convening Framework

  • Disaggregating data helps uncover and eliminate inequities. This example is data from Detroit Public Schools (DPS).  DPS is in the news often and cited as having dismal outcomes.  If we were to compare state data with DPS, does it really look dismal?2015-16 Graduation and Dropout Rates


Disaggregating by one level would uncover some inequities, but disaggregating by two levels shows areas that can and should be addressed.2015-16_Grad_DO_rate_DTW_M_F



We hope you’ve enjoyed this week of aea365 hosted by the DUP TIG.  We’d love to have you join us at AEA 2017 and throughout the year.

The American Evaluation Association is hosting the Disabilities and Underrepresented Populations TIG (DUP) Week. The contributions all week are focused on engaging DUP in your evaluation efforts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

6 thoughts on “DUP Week: Accessible Evaluation Techniques with June Gothberg”

  1. Good day from Kamloops, BC, June! It looks like you posted this blog a few years ago already, but clearly, your ideas are just as relevant today! Like a few others who have already indicated in this comment thread before me, I am enrolled in the Professional Master’s program at Queen’s University. One of our courses explores Program Evaluation and Design. I have completed a PED plan for a program in my school called CONNECT, which aims to address learner fatigue and improve attendance on Fridays while building core competency skills like communication, critical thinking, and social awareness. One of our assignments was to find a blog post of personal interest to our PED designs in AEA365 and to reach out to the author – so here I am, reaching out to you!

    I am currently teaching in an alternative high school with a vulnerable population of learners; I have a huge heart for inclusion and the fair representation of those with disabilities and underrepresented populations. Your blog shares so many nuggets of truths for me, like: “Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.
    – Embed the principals of Universal Design whenever and wherever possible.
    – Utilize trauma-informed practice.”
    It is so important to apply the principles of UDL in our classrooms, I hadn’t even really considered whether or not my evaluations methods in my PED plan were being developed with an inclusive lens! I am going to spend some time reexamining my plan with a UDL filter of accessibility.

    I was also drawn to your lesson learned, “Disaggregating data helps uncover and eliminate inequities.” Clearly, this careful separation of data ensures that representation is more equitable and forthcoming for all.

    A question I have is how would you suggest to give stakeholders choice about the subject, style of communication, or any say in the organization when evaluation design plans seem to be organized so far in advance of any actual evaluation procedure? With a UDL evaluation framework, would it be wise to prepare several data collection methods for each evaluation question, or do you just aim to have a variety of collection methods over all?

    Thank you so much for your post. It has added a new layer of consideration to my developing program evaluation plan.

    Respectfully yours,
    Lisa Bucher

  2. Hi June! My name is Nataly, and I am a Master of Education student at Queen’s University. I am currently wrapping up a course in Program Inquiry and Evaluation, and have been given the task to reach out to an author of the AEA365 community and comment on an article of interest.

    As a special needs teacher, I understand the importance of ensuring that every voice is heard, and included. Too often evaluations are created in a one-size-fits-all approach, and that is not something that is conducive to the special needs population. These students’ abilities range from mild to severe (in learning abilities, motor skills, language skills, socioemotional skills, etc.) and I believe this has to be taken into great consideration when setting up an evaluation for this population.

    I was very content to find your blog on this topic and to see the great work your team puts into ensuring that all stakeholders are properly represented within the evaluation process. Your “Lessons Learned” are wonderful takeaways to keep in mind when planning a program evaluation for those with special needs. I was particularly drawn to your point, “Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.”

    Do you believe that forming relationships with the stakeholders’ prior and during the evaluation process will benefit the outcomes, as you will be able to better understand the abilities of the individuals and therefore plan your evaluations accordingly? Or are past experiences what your team mostly relies on? I am curious as to how you prepare and plan for your program evaluations.

    Thank you!

  3. Laurel Charnetski

    Dear Dr. Gothberg,
    Thank you so much for your contribution and very timely as I am evaluating an elementary school program in which 50% of the participants are disabled. If I do not adequately ensure every voice is heard, I risk alienating 50% of my stakeholders. Trough reading your post I realize that I must create access and time to ensure all stakeholders are represented when doing my evaluation.I really appreciated your statement that we need to “increase authentic participation, voice, recommendations, and decision-making by engaging all types and levels of stakeholders in evaluation planning efforts.” A few participants in the program have expressive language difficulties and I would appreciate any suggestions you might have in regards to collecting meaningful input.There are many facets to consider when developing an evaluation that allows equitable access and representation and I appreciate that you and the DUP are focused in this area and the other contributors on A365 during the week have been terrific.
    All my best, Laurel Charnetski

  4. Hello Ms. Gothberg,

    My name is Alia Hamadani and I am pursing my graduate studies in the Professional Masters of Education program at Queen’s University in Ontario, Canada. I am currently enrolled in the Program Inquiry and Evaluation course and was asked to connect with the evaluation community. As I am new to the evaluation world I am intrigued with all it encompasses and am eager to learn from professionals like you. Your article really resonated with me because of the keen interests I have in creating a PE that is equitable and in favour of all its stakeholders. I find this a little challenging, as there are many factors to consider when doing so. I appreciate the “lessons learned” you have mentioned as they have shed some light onto how this can be done and what to consider. However some questions that come to mind are “how does one consider all stakeholders if their interests and needs are different? If evaluation is required to focus on intended users or primary users and to create goals and expected outcomes collaboratively based on these, can it be done effectively?

    You mention “Increase authentic participation, voice, recommendations, and decision-making by engaging all types and levels of stakeholders in evaluation planning efforts.” How do we encourage stakeholders to engage in evaluation? The “Doing the Work Together Rubric” was very thought provoking with the different levels of interaction and provided me with a valuable resource to refer to when creating my own PE. How do we determine which stakeholders are involved at each level?

    Your insight is greatly appreciated.

  5. Hey Jane Gothberg, I am currently taking my Masters’ of Education through Queens University, and my current assignment, is to find an article that inspires me and write a reply to the author.

    I was inspired by your article because of the importance you stress on ‘voice’ (this theme really stuck out to the most in your article) in terms of making sure that all voices are heard and that results from an evaluation are presented to all stakeholders.

    I have two questions/comments to make.

    1) Hearing all voices and including all stakeholders is important, but what do you do when time and resources are limited? How do you ensure that you are including all stakeholders ‘voices?

    2) I was wondering as well, how do you communicate to those who from different cultures, who might speak a different language or because of cultural nuances might not be as open to communicating criticisms?

    1. John,
      These are great questions and ones we discuss often in our TIG meetings. For me, planning is the most important step and the best evaluation planning happens during program planning. It’s difficult to generically answer your questions because there is no one-size-fits-all solution. A lot will depend on your context, location, etc. For the communication issue, if you have planned your evaluation during the program planning, you will have an opportunity to identify a solution. Often you can recruit a person trusted by the group of concern to either fully communicate or help you communicate. There are times when situations require us to depend on others.
      I am sorry you waited so long for a reply. I do not get notices when posts are replied to on this blog hosted by AEA, I just saw this today. I hope you did well on your assignment and are out changing the world now.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.