DUP Week: Accessible Evaluation Techniques with June Gothberg

Greetings, I am June Gothberg, Ph.D. from Western Michigan University, Chair of the Disabilities and Underrepresented Populations TIG and co-author of the Universal Design for Evaluation Checklist (4th ed.).   Historically, our TIG has been a ‘working’ TIG, working collaboratively with AEA and the field to build capacity for accessible and inclusive evaluation.  Several terms tend to describe our philosophy – inclusive, accessible, perceptible, voice, empowered, equitable, representative, to name a few.  As we end our week, I’d like to share major themes that have emerged over my three terms in TIG leadership.

Lessons Learned

  • Representation in evaluation should mirror representation in the program. Oftentimes, this can be overlooked in evaluation reports.  This is an example from a community housing evaluation.  The data overrepresented some groups and underrepresented others.

 HUD Participant Data Comparison

  • Avoid using TDMs.
    • T = tokenism or giving participants a voice in evaluation efforts but little to no choice about the subject, style of communication, or any say in the organization.
    • D = decoration or asking participants to take part in evaluation efforts with little to no explanation of the reason for their involvement or its use.
    • M = manipulation or manipulating participants to participate in evaluation efforts. One example was presented in 2010 where food stamp recipients were required to answer surveys or they were ineligible to continue receiving assistance.  The surveys included identifying information.
  • Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.
    • Embed the principals of Universal Design whenever and wherever possible.
    • Utilize trauma-informed practice.
  • Increase authentic participation, voice, recommendations, and decision-making by engaginge all types and levels of stakeholders in evaluation planning efforts. The IDEA Partnership depth of engagement framework for program planning and evaluation has been adopted in state government planning efforts across the United States.

 IDEA Partnership Leading by Convening Framework

  • Disaggregating data helps uncover and eliminate inequities. This example is data from Detroit Public Schools (DPS).  DPS is in the news often and cited as having dismal outcomes.  If we were to compare state data with DPS, does it really look dismal?2015-16 Graduation and Dropout Rates

 

Disaggregating by one level would uncover some inequities, but disaggregating by two levels shows areas that can and should be addressed.2015-16_Grad_DO_rate_DTW_M_F

 

 

We hope you’ve enjoyed this week of aea365 hosted by the DUP TIG.  We’d love to have you join us at AEA 2017 and throughout the year.

The American Evaluation Association is hosting the Disabilities and Underrepresented Populations TIG (DUP) Week. The contributions all week are focused on engaging DUP in your evaluation efforts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

4 thoughts on “DUP Week: Accessible Evaluation Techniques with June Gothberg”

  1. Hi June! My name is Nataly, and I am a Master of Education student at Queen’s University. I am currently wrapping up a course in Program Inquiry and Evaluation, and have been given the task to reach out to an author of the AEA365 community and comment on an article of interest.

    As a special needs teacher, I understand the importance of ensuring that every voice is heard, and included. Too often evaluations are created in a one-size-fits-all approach, and that is not something that is conducive to the special needs population. These students’ abilities range from mild to severe (in learning abilities, motor skills, language skills, socioemotional skills, etc.) and I believe this has to be taken into great consideration when setting up an evaluation for this population.

    I was very content to find your blog on this topic and to see the great work your team puts into ensuring that all stakeholders are properly represented within the evaluation process. Your “Lessons Learned” are wonderful takeaways to keep in mind when planning a program evaluation for those with special needs. I was particularly drawn to your point, “Don’t assume you know the backgrounds, cultures, abilities, and experiences of your stakeholders and participants. If you plan for all, all will benefit.”

    Do you believe that forming relationships with the stakeholders’ prior and during the evaluation process will benefit the outcomes, as you will be able to better understand the abilities of the individuals and therefore plan your evaluations accordingly? Or are past experiences what your team mostly relies on? I am curious as to how you prepare and plan for your program evaluations.

    Thank you!
    Nataly

  2. Laurel Charnetski

    Dear Dr. Gothberg,
    Thank you so much for your contribution and very timely as I am evaluating an elementary school program in which 50% of the participants are disabled. If I do not adequately ensure every voice is heard, I risk alienating 50% of my stakeholders. Trough reading your post I realize that I must create access and time to ensure all stakeholders are represented when doing my evaluation.I really appreciated your statement that we need to “increase authentic participation, voice, recommendations, and decision-making by engaging all types and levels of stakeholders in evaluation planning efforts.” A few participants in the program have expressive language difficulties and I would appreciate any suggestions you might have in regards to collecting meaningful input.There are many facets to consider when developing an evaluation that allows equitable access and representation and I appreciate that you and the DUP are focused in this area and the other contributors on A365 during the week have been terrific.
    All my best, Laurel Charnetski

  3. Alia Hamadani

    Hello Ms. Gothberg,

    My name is Alia Hamadani and I am pursing my graduate studies in the Professional Masters of Education program at Queen’s University in Ontario, Canada. I am currently enrolled in the Program Inquiry and Evaluation course and was asked to connect with the evaluation community. As I am new to the evaluation world I am intrigued with all it encompasses and am eager to learn from professionals like you. Your article really resonated with me because of the keen interests I have in creating a PE that is equitable and in favour of all its stakeholders. I find this a little challenging, as there are many factors to consider when doing so. I appreciate the “lessons learned” you have mentioned as they have shed some light onto how this can be done and what to consider. However some questions that come to mind are “how does one consider all stakeholders if their interests and needs are different? If evaluation is required to focus on intended users or primary users and to create goals and expected outcomes collaboratively based on these, can it be done effectively?

    You mention “Increase authentic participation, voice, recommendations, and decision-making by engaging all types and levels of stakeholders in evaluation planning efforts.” How do we encourage stakeholders to engage in evaluation? The “Doing the Work Together Rubric” was very thought provoking with the different levels of interaction and provided me with a valuable resource to refer to when creating my own PE. How do we determine which stakeholders are involved at each level?

    Your insight is greatly appreciated.

  4. Hey Jane Gothberg, I am currently taking my Masters’ of Education through Queens University, and my current assignment, is to find an article that inspires me and write a reply to the author.

    I was inspired by your article because of the importance you stress on ‘voice’ (this theme really stuck out to the most in your article) in terms of making sure that all voices are heard and that results from an evaluation are presented to all stakeholders.

    I have two questions/comments to make.

    1) Hearing all voices and including all stakeholders is important, but what do you do when time and resources are limited? How do you ensure that you are including all stakeholders ‘voices?

    2) I was wondering as well, how do you communicate to those who from different cultures, who might speak a different language or because of cultural nuances might not be as open to communicating criticisms?

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.