Hi! I’m Hildie Cohen, Research Director at NORC at the University of Chicago. Listening and pausing is a skill that I am constantly practicing in my work managing research and evaluation projects. I must listen to clients in order to design evaluation plans, reports, and deliverables that meet the goals of the project. I must listen to staff who are assigned to carry out the multitude of tasks involved in preparing, implementing, and analyzing the results of the evaluation. I must listen to stakeholders who will receive, read, or utilize the evaluation findings. And, I must listen to subject matter experts who provide advice on methodology, survey design, and analytics.
Lessons Learned:
But the most salient piece of wisdom I have received about listening was from conducting an evaluation of a curriculum designed to train future adolescent healthcare works to address substance abuse with adolescents. I was attending a workshop and a leader from a youth-service organization stated that when she designs a program or evaluation, she follows the rule, “Nothing about them, without them.” Since I have heard this phrase, it has changed the way that I approach evaluation work.
This rule of thumb applies both to designing an intervention and data collection. For me, this means providing a space for adolescents to test aspects of the intervention and give their feedback. On another project evaluating an HIV-prevention program for urban, black men who have sex with men, it meant inviting members of this community to pilot surveys, data collection methods, and trainings.
Planning an effective evaluation that will lead to high quality and representative results takes time. This planning takes even more time when I build in the steps to listen to and reflect upon the perspectives of all constituents involved. But, it is a necessary step in order to reduce bias in the research process and to ensure that we collect the most accurate information.
Rad Resource: Based on Patton’s Utilization-Focused Evaluation (2012), The Evaluation Center has a checklist for the steps of practicing this type of program evaluation. You can see the checklist at: https://wmich.edu/evaluation/checklists, Under step 3 they note that identifying, organizing and engaging primary intended users “optimizes the personal factor, which emphasizes that an evaluation is more likely to be used if intended users are involved in ways they find meaningful, feel ownership of the evaluation, find the questions relevant, and care about the findings.” I have found over and over again that trying to approach evaluation from a place of curiosity and willingness to learn how intended users of an intervention or findings feel about the work has led to useful information that advances the common goal of the project for all involved.
The American Evaluation Association is celebrating Arizona Evaluation Network (AZENet) Affiliate Week. The contributions all this week to aea365 come from our AZENet members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Hildie,
Your first paragraph resonated with me from my lens at work as an administrator of a school. I find so much of what I do involves working collaboratively with all stakeholders and effective communication plays a huge role in how I do this. It’s so easy to dive into a situation or “take over” with one’s ideas, but wise to take a pause and really zone in what someone is saying. Not just listening to the words but really understanding and appreciating the full picture involving identities, circumstance, lens, and even from a sense of finding ways to support or work together with actions that target key concerns.
As a student enrolled in a Masters Program, I am learning more about program evaluation from a theory lens and perspective that I can then re-evaluate my current and past experiences with evaluation. Much of what you have communicated with listening involves working with stakeholders collaboratively to develop better outcomes.
In thinking of a program evaluation design, I found Patton’s work interesting as he encourages evaluators to collaborate with those who make use of the evaluation. He recommends that they be engaged as “actively involving primary intended users, the evaluator is preparing the groundwork for use.” (Patton, 2008, Chapter 3) Your statement, “nothing about them, without them”, seems to work hand in hand with Patton. This statement has been said many times as we engage in Indigenous learning and education, for the very reason you have highlighted in terms of reducing bias. What are some resources or additional work you engage in to reduce bias in your work?
The site you have shared with checklists is interesting and will prove useful as I endeavor to design a program evaluation with the Utilization-Focused Evaluation in mind. Thank you for this.
Best,
Maryam
Patton, M.Q. (2008). Utilization-focused evaluation, 4th edition. Thousand Oaks, CA: Sage.
Hi Hildie,
Thank you for your wonderful article and wisdom on the importance of listening and pausing. These are two important actions that we often tend to throw aside as we become more knowledgeable and skilled in a certain field. Currently I am a high school vice principal living in China, as well as a Master’s of Education student at Queen’s University.
Throughout my education and teaching experience and program design, I certainly agree with your notion on the importance of listening to all those around you. Although you wrote your article on the importance of listening in program design, I can draw a lot of parallels to listening in program design and in teaching. In private school administration, listening to clients can be compared to listening to parents- They can have an important role in helping you meet the goals of your school or teaching for that year. The staff to listen to would be teachers, they are carrying out the tasks you designed, they do analysis and without them your school or program could not run. The stakeholders are the students, very valuable and needed, they take your program design and receive, read and utilize the data. Lastly, the subject matter experts could be your principal and other administrators and school board members, they provide advice on the actions and plan of your program. Each of these groups is very important in bringing success to a school or program design, in both, one must remember to listen and pause.
I also really thought the quote “Nothing about them, without them” was powerful and definitely should be followed with regards to program design, teaching, or running a school. It is clear that success of a program and a program evaluation is based on not alienating people and keeping them happy and supportive of all that is being done.
Let me know if you agree with some of what I have written!
Thanks for the great article,
Mitch
Dear Hildie,
Thank you so much for your insights on the importance of listening! I have been an elementary school principal for ten years and I would like to share with you why your article resonated with me so much. Firstly, two of your points really stood out for me: planning an effective evaluation requires listening and reflecting on the perspectives of the constituents involved and “evaluation is more likely to be used if intended users are involved in ways they find meaningful” and “feel ownership of the evaluation”.
In my role as administrator, what I find seems counter-intuitive, yet it is true: as my experience in the role has increased, I sometimes find that my listening skills have actually regressed. What I mean by this is that when a teacher or staff member conveys something to me, I have most likely already had a similar experience or I make an assumption about what they are saying or what they need. While I may be correct at times, often enough, I am not, and either way, I am formulating a response in my head (and sometimes conveying that response) rather than fully listening to the points being made. Especially during teacher performance appraisals, by listening to the teachers and viewing the situation from their perspective, our counsel can be more accurate and much more meaningful to improve future teacher performance. Further, sometimes the teachers just need to be heard and they do not even require a response! I have made it a point in recent years to make a more concerted effort in fully listening before responding and your article affirms this course of action for me! Again, thanks so much!
Michael
Just to add to my last comment on February 21, it was illuminating to read Dr. Richard Kreuger’s interview about listening carefully in interviews. Dr. Kreuger is a professor at the University of Minnesota and has written several books on Focus Groups. He discusses good practices on how to prepare quality focus interviews with a number of listening strategies and techniques. One of the most important strategies is to stay focused as an interviewer rather than be in passive listening mode. I believe this might be even more important during Covid-19. If the interviewer dresses in ‘interview’ clothes, creates open-ended questions, and listens carefully, then the interview will produce high quality results.
Reference: Elliot, Susan. (2013, August 29). ‘Dr. Richard Kreuger on Qualitative Listening’ Interview Retrieved from: https://betterevaluation.org
Hi Hildie,
I am a Queen’s student in the Professional Masters of Education program. At the moment, I am taking a course entitled “Program Inquiry and Evaluation.”
Your blog on the Power of Listening was thoughtful and informative. I often think that many evaluators or teachers, based on ‘years of experience,’ feel that they already know what they are doing and therefore do not need to listen to others. This is a bias in itself–confirmation bias. We definitely need to instead make the space and time to listen and learn from others.
I am curious. Is the listening mostly part of your data analysis or do you put listening in all parts of your evaluation?
Thanks again for your blog.
Kathryn
This statement, “Nothing about them, without them,” really resonates with me. Can you provide another link to the checklist? It appears to be broken.
Hi! I’ve updated the link in Hildie’s article. Here is it as well: https://wmich.edu/evaluation/checklists