Ed Eval TIG Week: COVID-19’s Disruptive Impact on K-12 Educational Evaluation by Carrie Wiley and Becky Dvorak

Hi there! We are Carrie Wiley and Becky Dvorak. We are senior researchers at the Human Resources Research Organization (HumRRO) and work on a variety of educational program and policy evaluations. While the long-term impact of the shutdown or modification of nearly all aspects of society is yet to be seen, the short-term disruption to K-12 education is already taking its toll. The decision for nearly all states to cease state accountability testing for the 2019-20 school year not only impacts students, but has rippling effects on school funding, teacher evaluation, and current program evaluations seeking to show impact on student achievement by meeting What Works Clearinghouse (WWC) standards and to reach Tier 1 or Tier 2 level of evidence as defined by the Every Student Succeeds Act (ESSA). We offer some thoughts for evaluators to ponder as they navigate through unchartered territory.

Lesson Learned: The issue: A changed metric

• A naturally occurring global event has impacted the evaluation outcomes for current rigorously- designed evaluations relying on student achievement data from spring 2020.
• Programs seeking to show impact on student achievement are now faced with the lack of a valid and reliable student achievement outcome measure.

Hot Tip: Can we use a different metric?

• One alternative is to use the results from a similarly reliable and valid measure of learning that were collected during the 2019-20 school year prior to the COVID-19 school closures. The likelihood of such measures existing are likely slim.

Hot Tips: Can we delay the analysis until 2021?

• Another alternative is to delay the evaluation outcome until the 2020-21 school year. Various factors impact this decision – funding, stakeholders, and proximity of the treatment. That is, if the treatment occurred during the 2019-20 school year, would you expect to see impacts a year later or would those impacts be diminished to the point that you would not be able to attribute the program to having a meaningful effect on learning? Similarly, if a treatment was intended to address grade specific skills, are the tested skills one year later still relevant to the treatment?
• At this point we are uncertain whether COVID-19 will continue to impact schools during the 2020-21 school year.

Hot Tip: Can we measure and account for the size of the disruption?

• We know the COVID-19 crisis has disrupted K-12 learning; we do not know the magnitude of that disruption. In the absence of a comparison group who was not disrupted, how can we measure the size of the disruption? If we can measure the size of the disruption, we might be able to statistically account for it in 2021.

Hot Tips: What can we do now?

• Stay calm and let the dust settle.
• Be strategic in your thinking and potential design revisions, but allow for flexibility and patience.
• Consider the evaluation goals and impact on resources.
• Communicate with your client and engage with the evaluation community to help develop innovative ideas for different outcome measures and ways to determine the impact of disruption.

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.