Hello AEA365! We are Bryan Hall, senior director of evaluation, and Danielle Fontaine, senior manager of evaluation at BellXcel (formerly BELL, Building Educated Leaders for Life). BellXcel offers solutions for under-resourced communities to create evidence-based summer and afterschool learning programs that address the opportunity and achievement gaps.
Historically, the focus of our internal program evaluation approach was measuring student-level academic and social-emotional outcomes. We engaged teachers through survey research, but often only around program engagement and satisfaction. After a review of our evaluation data and anecdotal evidence, we discovered that our summer program was possibly having an impact on school year teaching practices. This lead us to look outside of our theory of change to consider how our pre-program professional development and in-program coaching may be impacting long-term teaching practices. Through additional survey research and in-depth interviews a few months after the summer program ended, we learned that our summer program can transform and enhance teaching practices, mindset, and ultimately influence their daily school year practices.
Lesson Learned:
Engage educators in impact evaluation, using their feedback to understand teacher-level impact in addition to student-level impact. Viewing educators and instructional strategies as inputs and activities in influencing student outcomes was limiting our understanding of program impact. By moving beyond reflective survey questions about engagement and satisfaction, we developed a broader understanding of exactly how educators feel about the program experience and advanced our knowledge of how the program influences teachers professionally.
Hot Tips:
- Use an Outcomes Harvesting approach to understand trends outside of the standard outcomes. Looking at the full range of evidence, we found anecdotes of teachers who changed their practices because of the summer experience. With this as an outcome, we developed surveys and in-depth interviews to understand how exactly the summer experience was influencing these practices.
- Reimagine a theory of change, when the evidence is pointing to an important trend. Traditionally, our theory of change saw educators as inputs and professional development as activities. Based on new evidence, our summer intervention goes beyond a program for students and considers educators as direct participants in the intervention.
- Start with exploratory research. Beginning with a small-scale, no-cost survey in 2016, we intentionally and over time explored which areas should be further investigated using more complex higher-cost methods.
Rad Resources:
AEA365 blog post, Outcomes Harvesting Week: What is Outcome Harvesting? By blog series hosts, Barbara Klugman, Heather Britt, and Heidi Shaeffer. This was the first in a week-long series included great information about Outcomes Harvesting.
Allison Crean Davis’s article In Building an Evidence Base for Research, Start Small – & Follow the Train of Breadcrumbs Toward Larger Answers.
Lovely Dhillon and Sara Vaca’s paper on Refining Theories of Change in the Journal of MultiDisciplinary Evaluation.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Bryan and Danielle
Teachers ‘ classroom methods are one of my concerns in the school systems. I believe this is especially important for early childhood learners. Early learners need a strong foundation. It is encouraging that you are looking at interventions that will improve teachers’ practices.
Hello Sheila,
Thank you for your article. I am currently taking my Masters and I am enrolled in a course that focuses on Integrating Planning, Instruction, and Assessment. I am also in my first year of teaching and learning many things from my students and colleagues
I was recently tasked with finding a fun and interactive way for students to practice math at home. Every week my grade team meets to discuss ways we can improve our teacher. The past month we’ve been focusing on math in the classroom. We found that the kids were not doing their homework as much as we would like so we needed to find a way to make it more fun, interactive and engaging. My grade team and I decided to try a math platform called TVO mPower (https://mpower.tvo.org/educators/#/dashboard/home).
I am a first year teacher so I am very open to trying to things both in and out of the classroom. I’ve learned that sometimes things go well and other times they do not go according to plan. I am learning to take risks and go outside of my comfort zone. My using the mPower site first allows me to gain my own thoughts, opinion and perspective of the site. I am a leader in the sense that I will provide information to my grade team and see if the site is a good fir for what we need in the classroom and for our students.
Although we have just begun using the site, we can already see an increase in student engagement when it comes to math. I find my students to be more willing to take risks and make mistakes in their learning. In addition, we are constantly asking our students to provide feedback based on their experiences so far. Each teacher has anecdotal records of how students feel towards mPower and their level of satisfaction with the program to date. Finally, my team and I are always exploring new ways to teach math. We are fortunate enough to have the resources we need to try new things and we are willing to fail in order to succeed. Working collaboratively with my colleagues has been an essential component for student success. It has allowed me to share and gain valuable insights and also seek help when it is needed.
Thanks again for your insightful words.
I teach in rural in southern Saskatchewan and at times we feel that our school is under-resourced. This year we are focusing on increasing student literacy skills. Consequently, we are looking at programs within the school and evaluating their success. I enjoyed reading how your afterschool program impacted teaching practices. The teachers who participated in the program learned new evidence-based teaching practices, who in turn brought the practics into the school.
I am new to program evaluation and I am learning how program evaluation can provide information to improve school programs. I am evaluating the literacy program in my school and I am I was considering impact evaluation and I am working through my evaluation design. Your explanation is very helpful and the tips and resources you suggest will certainly help with the next stage of the assignment. Do think that outcome harvesting would work with a small population or is it better with larger populations?
Hello Bryan and Danielle,
My name is Meghan Filgate and I am a grade 2 teacher in Victoria, British Columbia. I really enjoyed reading this article and learning more about BellXcel. This is such a valuable and impactful program to have in place and I believe it would benefit many under-resourced communities.
In particular, I like that you pointed out beginning with a small-scale approach to evaluation. This allowed room to discover new aspects of the program that you were not aware of at the beginning, thus understanding the positive impact it was having on educators.
You mention that you did additional survey research and interviewing after the summer program ended. Now that you have seen different results than expected, would you consider a method of assessment throughout the summer versus once the program has ended? Maybe this is something that you already do and if so how?
As well, have you found that by engaging educators in impact evaluation like you mention, the buy-in from teachers has increased in the summer program?
Thank you for the well-articulated post. I really enjoyed reading about your program evaluation steps and I look forward to learning more about BellXcell.
It was refreshing to read that BellXcel has taken a look at the greater effects of their summer programming for educators. So often in educational programming and courses have I filled out surveys which ask for my feelings about content, instructor and course assignments, but I haven’t yet experienced the “Outcomes Harvesting approach” that you describe. As a teacher, I am constantly changing my teaching strategies and applying what I have learned to my classroom. When I feel that something I have learned could make my teaching better, I am eager to apply it to help my students access learning. I feel as though more educational institutions and professional development workshops should take the time to learn whether or not their information is being applied to a classroom. If it is, which one would hope as this should be its’ aim, then it can be said to be a useful course or program. However, if teachers are not using or applying that learning at all after a course or workshop is taken, then the educational institution or instructor should likely be re-evaluating their program. I appreciate learning about this approach as I had come to accept that all post-program surveys and data collection were fairly dull in their application and had no meaning in my life after the course was over but I hope to see more with an Outcome Harvesting Approach in the future.