Hi! I’m Lisa M. Jones, Chair of the PreK-12 Educational Evaluation Topical Interest Group (TIG) and Senior Managing Researcher/Evaluator at McREL International.
What a great time to kick off Teachers’ Week with an AEA365 blog!
Right now, our McREL team is evaluating a program spanning grades K-12. Our client wants to know if the program “works” for students within the school district. The Chief Academic Officer (CAO), who contracted our services, asked us to include the “usual suspects” (e.g., standardizes test scores, GPA, postsecondary enrollment) to capture program effects on students.
Evaluators know program effects rest in context: the program’s context, the schools’ context, the classrooms’ context. Test scores, GPA, and postsecondary enrollment data capture different variables but not context. While our “measurable” outcomes may show little to no effects, the stories we hear in focus groups and interviews capture the positive effects our current measures remain insufficient to obtain. As evaluators, we embrace what we learn from the stories people tell as credible evidence despite the argument verbally stated by less informed colleagues that qualitative data simply offers “anecdotal” information. Some days it seems as though we function in a world in which numbers matter more than educators’ and students’ lived experiences. We are asked to conduct implementation fidelity studies, for example, and reduce our findings to the percentage of users faithfully applying a strategy or program and offer conclusions and recommendations about how to improve implementation without describing the “why” it is unfaithfully implemented. How can we facilitate school district, state education agencies’, and policymakers’ understanding of the critical relationship between context and intended outcomes?
Refresh your understanding of AEA’s evaluator competencies with a laser focus on 3.0 Domain Context. It offers language to describe context.
The National Academies of Sciences recently released a report titled The Future of Education Research at IES: Advancing an Equity-Oriented Science, which acknowledges what evaluators already know: the parameters of credibility accepted by the Institute of Education Sciences (IES) – who often fund our program evaluations in schools –and other researchers have missed significant opportunities to understand how context affects how teachers teach and students learn. Our collective addiction to standardized test scores and grades as outcome measures leveraged in RCTs distracts us from discerning the positive effects programs and services do offer educators and students. Unfortunately, the highway is littered with programs, services, curricula that failed to meet ESSA tiers of evidence, yet yielded unmeasured or “less important” positive effects for educators and students. Evidence likely reflected in data derived from focus groups, interviews, and document reviews. In fact, the report recommends IES establish competitions and constitute a review panel that supports qualitative and mixed methods designs!
Our collective wisdom as evaluators acknowledged context for many decades. Context tells a story that GPA and scaled scores cannot as we see repeatedly in our work in schools, districts, and state agencies. I am thrilled IES reached out to the National Academies of Science for guidance in education research. As evaluators, we are uniquely situated to embrace opportunities exposing the positive effects of education programs and services not captured by the limitations of current outcome measures.
Check out the National Academies report at: https://doi.org/10.17226/26428. It’s an exciting time to be an educational evaluator! Happy trails!
Report citation: National Academies of Sciences, Engineering, and Medicine 2022. The Future of Education Research at IES: Advancing an Equity-Oriented Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/26428.
The American Evaluation Association is hosting PreK-12 Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our PreK-12 Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.