PreK-12 Ed Eval TIG Week: Validity, Context, and Circumstance by Sheila A. Arens

I’m Sheila A. Arens, Chair of the PreK-12 Educational Evaluation Topical Interest Group (TIG) and Executive Director of Research, Evaluation, and Technical Assistance at McREL. I am delighted to kick off this week of AEA365 blogs!

I spent considerable time in 2020/21 thinking about the nature of evaluation practice and the ways that context and circumstance shape not only what we are evaluating (and hence the warrantability of the claims we seek to make about value, merit or worth) but also how context and circumstance shape us as practitioners.

Lesson Learned:

Arguably, the past several years have been an enlightening journey for education evaluators; for the last 18 months, many of us have been forced to pivot evaluation projects in fairly significant, unexpected ways. To be sure, typical evaluation practice is rife with situations that demand flexibility, perseverance, and fortitude … evaluations rarely benefit from strict adherence to pre-planned designs and non-negotiable contracts. Almost invariably some brand of disruption occurs—some of these pose a lesser risk (staff turnover disrupts long-term data collection) and some pose a grand threat (a typhoon demolishes schools where evaluation was planned). The unanticipated twists and turns associated with much education evaluation renders it a bit too chaotic for some. But the past 18 months have shaken even the most stoic in our ranks. How can we move forward in our work?

Hot Tip:

Engage with fellow evaluators to explore ways the pandemic has disrupted their work. Seek out perspectives from AEA members in different TIGs.

Lesson Learned:

The past 18 months have afforded me a different window—one I didn’t choose, but one that is impossible to avoid and inextricably linked to others in my world … evaluation participants, fellow evaluators, coworkers, supervisees, and frankly members of civil society. I have learned how to problematize my own assumptions; it has been through dialogue with others—sometimes unexpected and seemingly tangential—that I’ve learned to listen harder, with more understanding, with more compassion.

I’ve always known contingency planning isn’t enough. Similarly, I’ve never believed in an airtight contract. Adherence to the standards of practice is critical, but never sufficient. What I didn’t realize is how unprecedented disruption to lives and livelihoods would fundamentally challenge me (and some of my colleagues) to think differently about what counts as credible evidence and, ultimately shape how I think about engaging with those who are participating in the programs and policies I evaluate. It is possible that none of this has (or should have) any bearing on the credibility of evidence or of claims, but it is equally possible that our community’s vision for what we can be as education evaluators has shifted. There will probably always be a place for the use of propositional logic and its rule-bound, formalistic, methods-driven ways of reaching evaluative assertions, but giving ascendancy to this thinking seems unwarranted. There is ample space for the inclusion of voices; for stories of joy and defeat; for multifaceted types and forms of evidence. As we reflect on what we hope to accomplish as a profession, we ought to be circumspect about assumptions, including those associated with evaluative validity.

Hot Tip:

Start a journal to track your assumptions … and question everything. Two years ago, state achievement tests were a given. Two years ago, teachers in the programs we are evaluating were worried about delivering the curriculum to their students in the prescribed manner. Two years ago the simple act of grocery shopping was simple. What do you assume “counts”—and how would “counting” those things resonate with the people most affected by your evaluation?

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

2 thoughts on “PreK-12 Ed Eval TIG Week: Validity, Context, and Circumstance by Sheila A. Arens”

  1. Lilian Garcia

    I agree that questioning everything will help to learn and understand information better. Especially having to write it down to remember to ask later.

  2. Hello Sheila,
    I found your post interesting. I think a good way of learning, as you mentioned before, can be by questioning things and also by allowing yourself to have curiosity. That way we can make correct changes in our patterns in order to improve ourselves.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.