Hi! I’m Lisa M. Jones, Chair of the PreK-12 Educational Evaluation Topical Interest Group (TIG) and Senior Managing Researcher/Evaluator at McREL International.
What a great time to kick off Teachers’ Week with an AEA365 blog!
Right now, our McREL team is evaluating a program spanning grades K-12. Our client wants to know if the program “works” for students within the school district. The Chief Academic Officer (CAO), who contracted our services, asked us to include the “usual suspects” (e.g., standardizes test scores, GPA, postsecondary enrollment) to capture program effects on students.
Lesson Learned
Evaluators know program effects rest in context: the program’s context, the schools’ context, the classrooms’ context. Test scores, GPA, and postsecondary enrollment data capture different variables but not context. While our “measurable” outcomes may show little to no effects, the stories we hear in focus groups and interviews capture the positive effects our current measures remain insufficient to obtain. As evaluators, we embrace what we learn from the stories people tell as credible evidence despite the argument verbally stated by less informed colleagues that qualitative data simply offers “anecdotal” information. Some days it seems as though we function in a world in which numbers matter more than educators’ and students’ lived experiences. We are asked to conduct implementation fidelity studies, for example, and reduce our findings to the percentage of users faithfully applying a strategy or program and offer conclusions and recommendations about how to improve implementation without describing the “why” it is unfaithfully implemented. How can we facilitate school district, state education agencies’, and policymakers’ understanding of the critical relationship between context and intended outcomes?
Hot Tip
Refresh your understanding of AEA’s evaluator competencies with a laser focus on 3.0 Domain Context. It offers language to describe context.
Lesson Learned
The National Academies of Sciences recently released a report titled The Future of Education Research at IES: Advancing an Equity-Oriented Science, which acknowledges what evaluators already know: the parameters of credibility accepted by the Institute of Education Sciences (IES) – who often fund our program evaluations in schools –and other researchers have missed significant opportunities to understand how context affects how teachers teach and students learn. Our collective addiction to standardized test scores and grades as outcome measures leveraged in RCTs distracts us from discerning the positive effects programs and services do offer educators and students. Unfortunately, the highway is littered with programs, services, curricula that failed to meet ESSA tiers of evidence, yet yielded unmeasured or “less important” positive effects for educators and students. Evidence likely reflected in data derived from focus groups, interviews, and document reviews. In fact, the report recommends IES establish competitions and constitute a review panel that supports qualitative and mixed methods designs!
Our collective wisdom as evaluators acknowledged context for many decades. Context tells a story that GPA and scaled scores cannot as we see repeatedly in our work in schools, districts, and state agencies. I am thrilled IES reached out to the National Academies of Science for guidance in education research. As evaluators, we are uniquely situated to embrace opportunities exposing the positive effects of education programs and services not captured by the limitations of current outcome measures.
Rad Resource
Check out the National Academies report at: https://doi.org/10.17226/26428. It’s an exciting time to be an educational evaluator! Happy trails!
Report citation: National Academies of Sciences, Engineering, and Medicine 2022. The Future of Education Research at IES: Advancing an Equity-Oriented Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/26428.
The American Evaluation Association is hosting PreK-12 Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our PreK-12 Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Hi Lisa,
Thank you for sharing your post! I am current pursuing my professional Master of Education and am taking a course in program evaluation. As a grade 2 teacher I was drawn into your article as it specifically related to evaluation in schools. As you state, context is very important. Data and numbers show one specific part of the bigger picture, of what is occurring in the classroom, school and community. Prior to becoming a teacher, I worked as a Youth Employment Counsellor, as a not-for-profit the programs I was running, and the scale I was able to run the programs at was up the Ministry and depended on the number of youth that ended up in, and stayed in jobs. Similar to schools, the context was not taken into context, what was evaluated on was just the two numbers, how many youths completed the program, and after completing the program how many youth ended up in jobs. However, if there were focus groups, or more in-depth opportunities for evaluation that included context youth who completed the program and then returned to education to complete get their high school diploma or attend college could also be recognized as “successes”.
You mention how as an evaluator part of your role is to essentially get to know the effects of education programs and services on our deeper level, using focus groups and interviews to further understand the context. In my course we have discussed collaborative evaluation, and how this allows for greater connection people, environments, circumstance as well as the conduction of integrative evaluations that bring organizations together (Shulha & Cousins., 1997). However, this swing to collaborative evaluation also brings into question an evaluators ability to remain unbiased as they grow closer to the program and individuals involved. I understand the conflict that occur when marking or evaluating a student when I am aware of the context behind for example why a student may not have done great on their math test, and balancing that line of still assigning the appropriate grade. My question, is when you do have the opportunity to dig deeper into context, are there certain steps, or recommendations you take or have found helpful in remaining unbiased?
Thank you for your time!
All the best,
Mary
References:
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208
Hi Lisa
I am completing a professional education degree program and presently my learning has focused on program inquiry and evaluation. I am a math teacher who would like to measure and describe outcomes for our department and board, and I found your discussion of context relevant while also very intriguing. In Ontario, Canada, we are undergoing a major shift within our Grade 9 math program – major changes to curriculum and pedagogy have been initiated in the effort to increase equity within the system. As a result of these changes, math teachers in my school have begun to focus more on growth and progress as indicators of achievement, instead of the traditional ‘benchmark’ approach that we have historically relied upon (and is quantifiable). Progress and growth are not widely accepted indicators of achievement, and this certainly is not something that is accurately represented by standardized tests. I however have made countless observations this year that speak to the amazing things that have occurred for my students because of the changes to instruction and assessment, in terms of growing student learning and increasing positive outcomes. When attempting to validate or show the merit of the grade 9 program however, these aspects are less visible to outsiders, and thus there exists difficulty in capturing this information by an evaluation employing traditional methods. As you state in your article, “While our “measurable” outcomes may show little to no effects, the stories we hear in focus groups and interviews capture the positive effects our current measures remain insufficient to obtain.” This summarizes the impasse I have encountered as a teacher who honors and promotes a growth mindset yet is required to consolidate learning with a standardized measure of achievement.
It is exciting to hear that the IES, which is the corresponding body to the Education Quality and Accountability Office (EQAO) here in Ontario, has recognized the lack of contextual information in their evaluations and how this might play a role in the accuracy of the information collected. I am curious and hopeful that this information attracts the attention of policy makers here in Ontario. I believe in mathematics education we need to be now, more than ever, innovative about assessment and how we measure achievement. Long gone are the days where we need people to “perform mathematics” with high degrees of accuracy. Math ‘performance’ however remains the focus of most lessons, and it is what we select and quantify for assessment, not just in our classrooms, but also in our efforts to evaluate our programs. I am so glad I came across the resource you shared as it serves as an important resource that substantiates the argument for change in how we evaluate accountability in math education.
Hi Lisa,
I really enjoyed your post, right from the beginning where you mentioned that your client wants to know if the “program works” for students in the district. I am taking a class for my master’s program and we’re learning about program evaluation, and this point immediately stuck out to me as one to dig deeper into. It makes me think about how all program evaluation use is political, and how the questions we choose to ask, and study can have dramatic differences in the outcomes of the study. So whether the “program works” is up for debate and having discussions and input on what that means is so important.
Agreeing on what the “program working” means will also influence the decisions around data gathering, what to look at, who to talk to, how to gather input, qualitative vs quantitative. It seems standard for the desire to look at standardized test scores, but I agree that this does not provide a complete picture. Like you said, context is key, and a program does not happen in a vacuum. The social theory of what is happening needs to be included, and the qualitative data around the stories and experiences of the program participants are important to include. To me, this is an example of a larger issue of what the goal of many of these programs are. Funding is tied to outcomes, but there is disagreement about what those outcomes should be, so I feel like discussions should be happening on a larger scale around this question.
I loved how you said that as evaluators, we are in a position to share the positives of some of these contextual factors that are outside of the standard data sets. I couldn’t agree more and thank you for sharing your insight!
Lindsay Workman