I am Betsy Olson, Extension Educator at the University of Minnesota Center for Youth Development. I have been working in youth development for very nearly 10 years and yet young people still surprise me every day.
Most recently, I was surprised at the depth of response I received from an open ended online survey question. After watching twenty 3-5 minute training videos we asked high school aged youth to use their own words to describe the videos. In the past when I have asked a similar question on an in-person survey form, the majority of the responses have been an empty space. Meanwhile the responses I received from the 113 young people who took this online survey were filled with interesting feedback and thoughtful responses. The average response was 12 words long. I did a happy dance in my cubicle. Then I asked myself why I rarely get this kind of feedback on surveys from youth in other contexts. In other words, what went right on this survey?
Lessons Learned:
After some thought I identified a few elements I think made this survey work:
- Each youth respondent had dedicated a significant amount of time to this training. They were invested in the training and therefore were willing to take more time and thought to provide their insights.
- They could do it on their own time. So the typical rush at the end of a training to get out the door was not a factor.
- The anonymity of an online survey provided an environment free from concerns about offending staff or the judgement of friends.
- We had trusted them to complete the training independently, indicating we would trust their feedback.
Hot Tip: When youth are interested in a topic don’t be afraid to include open-ended questions in your formative surveys.
Hot Tip: When engaging youth in online activities or training, gathering qualitative input is much easier than through an in-person survey.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Betsy,
I’m a student in a Master’s program in Canada and am currently taking a course in Program Inquiry and Evaluation. I was really interested to read your piece about youth and survey data as I’m in the process of finalizing a project on a program evaluation for a youth camp with determining the best options for data collection methods.
I was happy to see the high response rate from youth and though I was initially surprised at that, I think the lessons you outlined do help make sense of that. When someone is invested in a team, a project, an event, I think they are more apt to take the time to provide feedback, and one of the main points I think is the fact that you allowed them to provide this feedback on their own schedule. I’m dealing with youth a bit younger, and while I would like to be able to provide them the survey to do on their own, I’m not sure they would have that maturity level to complete it. Have you found any differences in responses between high school aged youth? I feel that grade 9 students may have a different response rate than those who are almost leaving in grade 12.
I appreciated the tip about not being afraid to include open-ended questions – as I always thought it would be easier to analyze questions that had the same answers across the board. From your perspective, though I do understand the value in receiving that feedback, how do you organize the comments? Is it easiest to try to generalize comments after into themes? I’m going to look more into this.
Some of the above are what I’m going to be looking at with my camp program evaluation and you have given me a lot to think about in terms of how I will move forward with the collection of responses from the youth.
Thanks so much,
Aimee
Hi Betsy,
As a teacher, I was curious to see what you had to say about getting good survey data from youth as this can be challenging at the best of times…. In saying that, it is important to remind myself that young people can be incredibly insightful, especially when they feel comfortable. I just wanted to say thank-you for highlighting some important aspects when working with young people.
The trust aspect of your survey is something that all evaluators need to consider. Many people view evaluation in a negative light and do not see it as a mechanism for change. Whether or not we realize it, we should recognize that we sometimes send both implicit and explicit messages when asking questions.
In your lessons learned section you identified some important points. Having the youth invested in the training makes them more willing to participate and that ownership piece is validated (Taut, 11). When people are personally attached to a program they may be more willing to invest the time in evaluation.
The anonymity of the survey probably did not make the youth feel as though they were being judged when they were answering the questions. They did not have to worry about whether or not they were giving you the answer you wanted to hear. With technology constantly changing some youth feel more comfortable presenting their thoughts and ideas on online platforms. As Carol Weiss suggests evaluators should chose to use evaluation to take clients’ and interests into account and address their concerns (Weiss, 27). As an educator I need to remind myself to give students time to process information and reconsider how they should share it.
I think it is wonderful that you encourage your readers to not fear open-ended questions. People tend to shy away from qualitative questions for a variety of reasons. Weiss reminds us that “What evaluation finds out depends o what the evaluator focusses on and measures and how she measures them.” (Weiss, 31). Perhaps we should be more willing to take risks, who knows what we will find.
Thank-you for sharing.
Steph
Alkin, M. C., and Taut, S. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1-12.
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19, 21-33https://aea365.org/blog/category/youth-focused-evaluation/
Hello Betsy Olson,
I enjoyed reading your article and I am happy for your success with the open-ended survey questions. I know that for me, when I am done a training session or professional development the last thing I want to complete, at the end, is open-ended survey questions. I appreciated that the students took the time to complete the questions in an in-depth, thoughtful manner. I think you had success with the questioning, as you mentioned, because of the trust that was established and that the students were able to complete it on their own time. I was surprised that you didn’t include the fact that the survey was online as being one of the reasons it worked. I think that with that generation being so technologically inclined, having the survey in an online format was a big strength. Would it have been as successful if it had been a written survey, to be filled out and returned at a later date? As I read the article, I also wondered if open-ended survey questions would work better if the survey were given at the beginning of training sessions. That way the participants could be critically thinking about the training and making notes as it progresses. I often think of things to say/write during training, but then when it comes time to complete the survey I have forgotten my points. Thanks for the interesting read!
Hi Betsy. I enjoyed reading your post. I’m a Teacher and I feel in many cases getting our youth or students to answer surveys for data is nearly impossible. In fact I feel it is the same with adults. I’m in the process of gathering data for a process of evaluation with in my school. I have been trying to have feedback by having face to face conversations, and I’m finding I’m not getting as much information as I would like.
I will follow your hot tips. Engaging my focus group into meaningful online surveys. The other point is to create open ended questions. Perhaps I’m looking into a big picture with huge expectations and I should be focusing on my main topic. Less sometimes is more!
Thank you again for your post.
Alina