Teachers Are Important Participants in Student-centered Interventions, Not Just Inputs by Bryan Hall and Danielle Fontaine

Hello AEA365! We are Bryan Hall, senior director of evaluation, and Danielle Fontaine, senior manager of evaluation at BellXcel (formerly BELL, Building Educated Leaders for Life). BellXcel offers solutions for under-resourced communities to create evidence-based summer and afterschool learning programs that address the opportunity and achievement gaps.

Historically, the focus of our internal program evaluation approach was measuring student-level academic and social-emotional outcomes.  We engaged teachers through survey research, but often only around program engagement and satisfaction.   After a review of our evaluation data and anecdotal evidence, we discovered that our summer program was possibly having an impact on school year teaching practices.  This lead us to look outside of our theory of change to consider how our pre-program professional development and in-program coaching may be impacting long-term teaching practices.  Through additional survey research and in-depth interviews a few months after the summer program ended, we learned that our summer program can transform and enhance teaching practices, mindset, and ultimately influence their daily school year practices.

Visual representation of BellXcel’s professional development model and outcomes
Visual representation of BellXcel’s professional development model and outcomes. (Source: Bellwether Education Partners)

Lesson Learned:

Engage educators in impact evaluation, using their feedback to understand teacher-level impact in addition to student-level impact. Viewing educators and instructional strategies as inputs and activities in influencing student outcomes was limiting our understanding of program impact. By moving beyond reflective survey questions about engagement and satisfaction, we developed a broader understanding of exactly how educators feel about the program experience and advanced our knowledge of how the program influences teachers professionally.

Hot Tips:

  1. Use an Outcomes Harvesting approach to understand trends outside of the standard outcomes. Looking at the full range of evidence, we found anecdotes of teachers who changed their practices because of the summer experience. With this as an outcome, we developed surveys and in-depth interviews to understand how exactly the summer experience was influencing these practices.
  2. Reimagine a theory of change, when the evidence is pointing to an important trend. Traditionally, our theory of change saw educators as inputs and professional development as activities. Based on new evidence, our summer intervention goes beyond a program for students and considers educators as direct participants in the intervention.
  3. Start with exploratory research. Beginning with a small-scale, no-cost survey in 2016, we intentionally and over time explored which areas should be further investigated using more complex higher-cost methods.

Rad Resources:

AEA365 blog post, Outcomes Harvesting Week: What is Outcome Harvesting? By blog series hosts, Barbara Klugman, Heather Britt, and Heidi Shaeffer. This was the first in a week-long series included great information about Outcomes Harvesting.

Allison Crean Davis’s article In Building an Evidence Base for Research, Start Small – & Follow the Train of Breadcrumbs Toward Larger Answers.

Lovely Dhillon and Sara Vaca’s paper on Refining Theories of Change in the Journal of MultiDisciplinary Evaluation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

2 thoughts on “Teachers Are Important Participants in Student-centered Interventions, Not Just Inputs by Bryan Hall and Danielle Fontaine”

  1. Hello Bryan and Danielle,

    My name is Meghan Filgate and I am a grade 2 teacher in Victoria, British Columbia. I really enjoyed reading this article and learning more about BellXcel. This is such a valuable and impactful program to have in place and I believe it would benefit many under-resourced communities.

    In particular, I like that you pointed out beginning with a small-scale approach to evaluation. This allowed room to discover new aspects of the program that you were not aware of at the beginning, thus understanding the positive impact it was having on educators.

    You mention that you did additional survey research and interviewing after the summer program ended. Now that you have seen different results than expected, would you consider a method of assessment throughout the summer versus once the program has ended? Maybe this is something that you already do and if so how?

    As well, have you found that by engaging educators in impact evaluation like you mention, the buy-in from teachers has increased in the summer program?

    Thank you for the well-articulated post. I really enjoyed reading about your program evaluation steps and I look forward to learning more about BellXcell.

  2. Kirsten Andrews

    It was refreshing to read that BellXcel has taken a look at the greater effects of their summer programming for educators. So often in educational programming and courses have I filled out surveys which ask for my feelings about content, instructor and course assignments, but I haven’t yet experienced the “Outcomes Harvesting approach” that you describe. As a teacher, I am constantly changing my teaching strategies and applying what I have learned to my classroom. When I feel that something I have learned could make my teaching better, I am eager to apply it to help my students access learning. I feel as though more educational institutions and professional development workshops should take the time to learn whether or not their information is being applied to a classroom. If it is, which one would hope as this should be its’ aim, then it can be said to be a useful course or program. However, if teachers are not using or applying that learning at all after a course or workshop is taken, then the educational institution or instructor should likely be re-evaluating their program. I appreciate learning about this approach as I had come to accept that all post-program surveys and data collection were fairly dull in their application and had no meaning in my life after the course was over but I hope to see more with an Outcome Harvesting Approach in the future.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.