Hello, my name is Stella SiWan Zimmerman, and I am the Founder and President of ACET, Inc. in Minneapolis, MN.
A year ago, ACET was selected to evaluate the Hmong American Partnership’s 21st Century Community Learning grant from the Minnesota Department of Education. The Hmong American Partnership collaborates with the Community School of Excellence to implement out-of-school time academic and enrichment programming aimed to “help students make important connections among their studies, curiosities, their passions, and the skills they need to become critical thinkers and productive members of society.”
As part of the evaluation, ACET conducted multiple observations using the Youth Program Quality Assessment (YPQA) to report on four domains: 1. Safe Environment, 2. Supportive Environment, 3. Interaction, and 4. Engagement. As an endorsed external assessor, we were able to follow the YPQA Assess-Plan-Improve sequence to analyze data to provide helpful information for program effectiveness.
“Too often, youth programs and evaluation plans do not include youth in shared decisions, especially with quality improvement initiatives.”
-Stella SiWan Zimmerman
Following the observations, ACET met with program and school staff to review the results and discuss next steps to strengthen programming and for engaging youth to help improve program quality. Too often, youth programs and evaluation plans do not include youth in shared decisions, especially with quality improvement initiatives. To involve young people, I offer 3 hot tips based on prior experience engaging youth with program evaluation:
Hot Tip #1: Examine opportunities to include youth before building an action plan as part of quality improvement.
The YPQA, for example, includes many areas focused on including youth voices and identifying ways for youth to make shared decisions, mentor other youth, problem solve, and lead activities. To better support next steps with YPQA results, recruit youth and help them to familiarize themselves with what expectations the school has for the programs it offers. This can help promote their support and understanding of before- and after-school activities because they understand how the activities relate to school goals.
Hot Tip #2: Have your data collection tools reviewed by youth.
Initial questions identified for a survey or script will likely need to be modified using language that youth use. The process of gathering this feedback can be facilitated by the evaluator in a small group setting. The evaluator can ask youth their thoughts on certain words used, phrases, as well as interpretive meaning.
Hot Tip #3: Sustain youth involvement in evaluation by developing relationships with them.
Too often, evaluations involve youth at only certain points in the evaluation process. By identifying ways to involve youth throughout the evaluation cycle, you’ll have more buy-in and enthusiasm from youth—and a stronger evaluation.
Hot Tip for visitors to Minnesota for Evaluation 2019:
Minnesota is home to many professional sports teams such as Minnesota United, Minnesota Lynx, Minnesota Twins, and Minnesota Vikings. We also have the Minnesota RollerGirls (roller derby), and the minor league Saint Paul Saints baseball team.
We’re looking forward to the fall and the Evaluation 2019 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.
Thanks for this post! Just getting to this now. I hear from clients often that it’s a huge ask for youth to involve them in anything outside of a survey- and also concerned about parent consent, etc. Also a survey is easy to get broad feedback but focus groups or other activities take a lot of time and a specific skill set. Any recommendations or tips you can share on how to effectively recruit youth and manage parents?
Nitya,
Thank you for your thoughtful questions. You’re right: It’s a huge ask to involve youth in anything outside of a survey if youth typically are not active in leadership roles. In those cases, encouraging youth to participate in a survey may be enough.
However, different programs have different levels of youth involvement. Some that have high youth engagement may find it quite easy to include youth in evaluation tasks. In working with program staff, identify what skill sets youth could gain from the evaluation (e.g., how to interview, how to write questions for a survey, how to create tables and visuals, how to tabulate responses from the survey). Would these skill sets be something that program would like to incorporate as part of enrichment activities for youth?
As evaluators, we work with program staff to talk about ways we can include data collection for youth and gaining parental support through programming. There are many opportunities to teach young people skills about evaluation through the process.
We also gain parental support by working closely with program staff. Parental support is very important, and program staff already have contact with parents of youth in the program. Together, we work out how to receive parental support by partnering with program staff.
Best wishes as you effectively recruit youth and gain parent consent.
Sincerely,
Stella
Hello,
In any youth engagement effort, young people should be involved in design and decision-making from the beginning. As one young woman from Advocates for Youth said, “Young people are the experts of their own experience.” Even the most in-tune adult allies cannot understand youth needs as well as young people themselves.
Hi Stella,
Thank you for this post. It caught my eye immediately by the title and I love all things that include youth’s voices and giving them a positive platform to share their opinions and help them bring about positive changes in programs they are involved in. I am currently a middle years teacher and a masters student and am completing a course on Program Inquiry and Evaluation. I found your post so helpful in many areas. Before taking this course I had not thought of program evaluation at all – it was always to get a grade for reporting periods and for feedback to students. Your post helped me see even more how education can use the program evaluation and our students input in so many valuable ways – the one that stuck out for me was in language used – so important!
From reading one of the posts above, May talked about the trust factor and talking to teachers and staff at the school about the students that should be involved in the building of an action plan, having them review the tools and developing relationships with them to sustain their involvement in the evaluation – I completely agree with this – there are students who would understand and trust this process when given the opportunity to be involved and there are others that would appear to be, but would not buy-in and have negative affects on the evaluation process. From this, I had a question about student involvement – have you ever been able to use a student that seems to be negative and apprehensive about what your team is trying to do and have it positively affect your evaluation planning? If so, has the student been able to see the positive benefits of what you are providing to their program, once the evaluation process is complete?
Thank you so much,
Katharine Kerr
Hi Katharine,
I always get excited when I hear about people like yourself who see how program evaluation with student input can be valuable. Thank you for sharing about your profession and coursework.
In looking at your question about involving a student who may seem negative and apprehensive, I connected with our communications editor about this. She said that often individuals who are negative and/or apprehensive have questions or concerns. Taking the time to listen to the individual and get his/her ideas often helps. For some individuals, you may need to do this many times throughout the process.
By the end, some come around, and you see positive benefits. Others may still be reluctant and negative, but if a relationship has been built, you often will notice some subtle, positive changes, such as the person drawing closer to you and/or others.
I hope you find this helpful.
Sincerely,
Stella
Hi Stella,
You are absolutely right about “youth can mentor each other” as youth view each other on the same level of playing field as peers or mentors. The notion of trust can be solidified through the engagement of common discussion topics and activities. Say an evaluator wishes to form a respectable bond with youth, then the evaluator will have to invest time and put in effort into seeking advice from a teacher, a parent, someone who may know that young person on a personal level. Making a personal connection is key in building a trusting relationship between people, isn’t?
Hope this helps!
May Ng
Hi Stella,
When I read the title of your post, I was immediately drawn to your topic on “Engaging Youth to Improve Program Quality”. You are absolutely right about involving youth in shared decisions especially with quality improvement initiatives as they are the ones who will benefit from participating in the out-of-school extra-curricular activities. Since they are the target participants, they should be involved from the onset of the program to get buy-in and become willing participants throughout the evaluation process.
Your Hot Tip # 1 makes a lot of sense where youth should have a voice in the program.
Moreover, when you recruit them to become peer counsellors/mentors for other youths, these individuals will learn leadership skills and have a sense of ownership towards the program which in turn will help to improve overall school culture and meet school goals.
Your Hot Tip # 2 on having data collection tools reviewed by youth is a great tip because youth has their own lingo & jargon that they use to communicate with each other; take a look on how they utilize social media. In small group settings, they will be more open to speak to the program evaluator about their thoughts once “trust” has been established.
If you want a stronger evaluation, your Hot Tip # 3 is bang on! By developing relationships with youth, you can sustain their involvement in evaluation. As a program evaluator, if you can make authentic connections with youth, and they see you as an ally, then they will engage in activities with an open mindset and a growth mindset.
Final thought, certain youth will be easier to approach than others based on their own experiences working with adults (program evaluators); I’m alluding to the trust factor. I would suggest asking teachers and counsellors at school for suggestions on whom to approach first for buy-in.
Thanks so much for your thoughtful and engaging post!
May Ng
Thanks May, for your helpful comments. You’ve added even more important information and tips to the blog about engaging youth to improve program quality.
I was connecting with my staff regarding your comments, and you’ve raised a great issue around the trust factor. How do evaluators work to build trust among youth? I would love to hear your thoughts on this as well. We’ve also found that, depending on the experience of youth and evaluation, youth can mentor each other in youth-involved evaluation as well.
Have a great week!
Stella
Hi Stella,
Thank you for posting the thoughtful hot tips on engaging youth in program evaluation. Your three tips are important reminders to program evaluators to work with youth at all stages of the process, not simply sending participants a survey at the end of the program.
In particular, I can see how it would be helpful to have feedback from the target audience(youth) on survey question language to ensure it resonates properly.
Can you share an example of what survey questions in youth-friendly language looks like to give us a sense of how questions might be modified as well?
Thanks for sharing!
Myke
Thanks Myke, for posting your thoughtful comments. Developing survey questions in youth-friendly language is key. Jolene, our Communications Editor, monitors the readability of a survey using a readability formula. One helpful readability formula she uses is the Flesch-Kincaid Grade Level formula. A grade 8-level can be read by 80% of adults, but that’s often too high for certain young people. We often aim to have a grade level around two grades lower than the age group we’re evaluating since there will be a diversity of reading levels.
We also know that it’s important to test survey questions for young people before you finalize the survey. An evaluator can sit down with a few young people and go through the survey together, asking for how the youth interpret each question. (Sometimes young people interpret a question differently than an adult would and some won’t answer a survey question if they don’t understand the question.) This helps to ensure that the survey language resonates with young people.
Sincerely,
Stella