I’m Sheila A. Arens, Chair of the PreK-12 Educational Evaluation Topical Interest Group (TIG) and Executive Director of Research, Evaluation, and Technical Assistance at McREL. I am delighted to kick off this week of AEA365 blogs!
I spent considerable time in 2020/21 thinking about the nature of evaluation practice and the ways that context and circumstance shape not only what we are evaluating (and hence the warrantability of the claims we seek to make about value, merit or worth) but also how context and circumstance shape us as practitioners.
Lesson Learned:
Arguably, the past several years have been an enlightening journey for education evaluators; for the last 18 months, many of us have been forced to pivot evaluation projects in fairly significant, unexpected ways. To be sure, typical evaluation practice is rife with situations that demand flexibility, perseverance, and fortitude … evaluations rarely benefit from strict adherence to pre-planned designs and non-negotiable contracts. Almost invariably some brand of disruption occurs—some of these pose a lesser risk (staff turnover disrupts long-term data collection) and some pose a grand threat (a typhoon demolishes schools where evaluation was planned). The unanticipated twists and turns associated with much education evaluation renders it a bit too chaotic for some. But the past 18 months have shaken even the most stoic in our ranks. How can we move forward in our work?
Hot Tip:
Engage with fellow evaluators to explore ways the pandemic has disrupted their work. Seek out perspectives from AEA members in different TIGs.
Lesson Learned:
The past 18 months have afforded me a different window—one I didn’t choose, but one that is impossible to avoid and inextricably linked to others in my world … evaluation participants, fellow evaluators, coworkers, supervisees, and frankly members of civil society. I have learned how to problematize my own assumptions; it has been through dialogue with others—sometimes unexpected and seemingly tangential—that I’ve learned to listen harder, with more understanding, with more compassion.
I’ve always known contingency planning isn’t enough. Similarly, I’ve never believed in an airtight contract. Adherence to the standards of practice is critical, but never sufficient. What I didn’t realize is how unprecedented disruption to lives and livelihoods would fundamentally challenge me (and some of my colleagues) to think differently about what counts as credible evidence and, ultimately shape how I think about engaging with those who are participating in the programs and policies I evaluate. It is possible that none of this has (or should have) any bearing on the credibility of evidence or of claims, but it is equally possible that our community’s vision for what we can be as education evaluators has shifted. There will probably always be a place for the use of propositional logic and its rule-bound, formalistic, methods-driven ways of reaching evaluative assertions, but giving ascendancy to this thinking seems unwarranted. There is ample space for the inclusion of voices; for stories of joy and defeat; for multifaceted types and forms of evidence. As we reflect on what we hope to accomplish as a profession, we ought to be circumspect about assumptions, including those associated with evaluative validity.
Hot Tip:
Start a journal to track your assumptions … and question everything. Two years ago, state achievement tests were a given. Two years ago, teachers in the programs we are evaluating were worried about delivering the curriculum to their students in the prescribed manner. Two years ago the simple act of grocery shopping was simple. What do you assume “counts”—and how would “counting” those things resonate with the people most affected by your evaluation?
The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.
Hi Sheila,
Thanks for sharing a very accurate description of evaluation and education over the last few months. There have been many changes and a lot of thinking on our feet, pivoting instruction and assessment which also leads to changes in our evaluations. I am currently completing a course on program evaluation and have recently learned of some of the struggles that occur daily let alone during a pandemic.
I appreciated how you discussed that this has been a situation that has been difficult and altering for everyone. However, what stood out the most to me in this post was when you said that “I’ve learned to listen harder, with more understanding, with more compassion.”. This is such a great reminder that we are all facing challenges, especially in our places of work. It reminded me of the Weiss-Patton debate, in which Weiss believed that evaluation should not be collaborative but have credibility, and Patton believed that evaluators should be involved in making decisions among the stakeholders (Shulha and Cousins, 1997). This makes me wonder how evaluation could be different if we were able to approach inclusion with more understanding and compassion. This past year has presented many challenges, I am curious to see what effect this will have on how we conduct classroom education but also how we conduct evaluations.
This year was a year full of disruptions. However, there is almost always disruption in education and evaluations. I think this shows that in evaluations we need to be critical thinkers and allow space for understanding, and to ensure we are not jumping to assumptions or holding bias. I like your idea of keeping track of a journal, reflection is a practice that I typically use with teaching that could be very beneficial in evaluation as well.
Thanks for sharing such an insightful post!
Ivy
References
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
I agree that questioning everything will help to learn and understand information better. Especially having to write it down to remember to ask later.
Hello Sheila,
I found your post interesting. I think a good way of learning, as you mentioned before, can be by questioning things and also by allowing yourself to have curiosity. That way we can make correct changes in our patterns in order to improve ourselves.