AZENet Week: The Power of Listening in Evaluation by Hildie Cohen

Hi! I’m Hildie Cohen, Research Director at NORC at the University of Chicago. Listening and pausing is a skill that I am constantly practicing in my work managing research and evaluation projects. I must listen to clients in order to design evaluation plans, reports, and deliverables that meet the goals of the project. I must listen to staff who are assigned to carry out the multitude of tasks involved in preparing, implementing, and analyzing the results of the evaluation. I must listen to stakeholders who will receive, read, or utilize the evaluation findings. And, I must listen to subject matter experts who provide advice on methodology, survey design, and analytics.

Lessons Learned:

But the most salient piece of wisdom I have received about listening was from conducting an evaluation of a curriculum designed to train future adolescent healthcare works to address substance abuse with adolescents. I was attending a workshop and a leader from a youth-service organization stated that when she designs a program or evaluation, she follows the rule, “Nothing about them, without them.” Since I have heard this phrase, it has changed the way that I approach evaluation work.

This rule of thumb applies both to designing an intervention and data collection. For me, this means providing a space for adolescents to test aspects of the intervention and give their feedback. On another project evaluating an HIV-prevention program for urban, black men who have sex with men, it meant inviting members of this community to pilot surveys, data collection methods, and trainings.

Planning an effective evaluation that will lead to high quality and representative results takes time. This planning takes even more time when I build in the steps to listen to and reflect upon the perspectives of all constituents involved. But, it is a necessary step in order to reduce bias in the research process and to ensure that we collect the most accurate information.

Rad Resource: Based on Patton’s Utilization-Focused Evaluation (2012), The Evaluation Center has a checklist for the steps of practicing this type of program evaluation. You can see the checklist at: https://wmich.edu/evaluation/checklists, Under step 3 they note that identifying, organizing and engaging primary intended users “optimizes the personal factor, which emphasizes that an evaluation is more likely to be used if intended users are involved in ways they find meaningful, feel ownership of the evaluation, find the questions relevant, and care about the findings.” I have found over and over again that trying to approach evaluation from a place of curiosity and willingness to learn how intended users of an intervention or findings feel about the work has led to useful information that advances the common goal of the project for all involved.

The American Evaluation Association is celebrating Arizona Evaluation Network (AZENet) Affiliate Week. The contributions all this week to aea365 come from our AZENet members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

5 thoughts on “AZENet Week: The Power of Listening in Evaluation by Hildie Cohen”

  1. Dear Hildie,

    Thank you so much for your insights on the importance of listening! I have been an elementary school principal for ten years and I would like to share with you why your article resonated with me so much. Firstly, two of your points really stood out for me: planning an effective evaluation requires listening and reflecting on the perspectives of the constituents involved and “evaluation is more likely to be used if intended users are involved in ways they find meaningful” and “feel ownership of the evaluation”.

    In my role as administrator, what I find seems counter-intuitive, yet it is true: as my experience in the role has increased, I sometimes find that my listening skills have actually regressed. What I mean by this is that when a teacher or staff member conveys something to me, I have most likely already had a similar experience or I make an assumption about what they are saying or what they need. While I may be correct at times, often enough, I am not, and either way, I am formulating a response in my head (and sometimes conveying that response) rather than fully listening to the points being made. Especially during teacher performance appraisals, by listening to the teachers and viewing the situation from their perspective, our counsel can be more accurate and much more meaningful to improve future teacher performance. Further, sometimes the teachers just need to be heard and they do not even require a response! I have made it a point in recent years to make a more concerted effort in fully listening before responding and your article affirms this course of action for me! Again, thanks so much!

    Michael

  2. Kathryn Baker

    Just to add to my last comment on February 21, it was illuminating to read Dr. Richard Kreuger’s interview about listening carefully in interviews. Dr. Kreuger is a professor at the University of Minnesota and has written several books on Focus Groups. He discusses good practices on how to prepare quality focus interviews with a number of listening strategies and techniques. One of the most important strategies is to stay focused as an interviewer rather than be in passive listening mode. I believe this might be even more important during Covid-19. If the interviewer dresses in ‘interview’ clothes, creates open-ended questions, and listens carefully, then the interview will produce high quality results.
    Reference: Elliot, Susan. (2013, August 29). ‘Dr. Richard Kreuger on Qualitative Listening’ Interview Retrieved from: https://betterevaluation.org

  3. Hi Hildie,
    I am a Queen’s student in the Professional Masters of Education program. At the moment, I am taking a course entitled “Program Inquiry and Evaluation.”

    Your blog on the Power of Listening was thoughtful and informative. I often think that many evaluators or teachers, based on ‘years of experience,’ feel that they already know what they are doing and therefore do not need to listen to others. This is a bias in itself–confirmation bias. We definitely need to instead make the space and time to listen and learn from others.
    I am curious. Is the listening mostly part of your data analysis or do you put listening in all parts of your evaluation?
    Thanks again for your blog.
    Kathryn

  4. This statement, “Nothing about them, without them,” really resonates with me. Can you provide another link to the checklist? It appears to be broken.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.