My name is Karen Kirkhart, and I am now an elder in the evaluation profession. I was active in both Evaluation Network and the Evaluation Research Society before they merged to form AEA. I have been active in AEA since its founding, and I served as President in 1994. I have spent decades pondering two related topics in evaluation: validity and influence. Creating accurate, trustworthy understandings, and taking action to make a difference. We evaluators have always wanted to use our craft to inform decisions and advance social betterment. In fact, the Use TIG was one of the original topical interest groups of the AEA, and the wisdom our TIG has never been more relevant than now.
The current political climate disrupts the connection between evidence and action. Truth, beauty, and justice—the parameters of validity identified by Ernie House in his 1980 book, Evaluating with validity—are ignored. Power, rather than evidence, drives actions.
But actions have consequences, and these consequences demand attention, as do the consequences of inaction. Fortunately, evaluators have given a great deal of attention to use and influence over the years, and the framework of influence is also well-suited to identifying/examining consequences. Understanding consequences, in turn, can assist in determining the validity of the actions themselves.
At the 2021 [online] AEA conference, I presented an expert lecture, titled “Evaluation Influence, Revisited,” and I described four parameters of influence—SOURCE, INTENTION, TIME, and LOCATION. Today, I see that these parameters of influence can not only help track the consequences of our work as evaluators, but also, they can be applied to critically examine consequences of actions beyond those based on evaluation.
- SOURCE refers to where the chain of influence begins—in the process of evaluation itself or from the results reported. Applied to consequences, it calls us to retrace the events and circumstances leading up to the action(s) in question.
- INTENTION demands that both intended and unintended consequences of actions be considered, just as intended and unintended outcomes are included in evaluation.
- TIME reminds us to take a historical perspective on consequences. When do they appear and what is their duration? Time cautions us to take the long view and avoid drawing hasty conclusions.
- LOCATION calls attention to where consequences appear. Location may be physical, structural, or cultural. Consequences should be examined for various communities. Location sharpens attention to where one is looking and not looking for consequences.
And importantly, Cronbach reminds us that the overarching concern is for validity:
“The bottom line is that validators [evaluators] have an obligation to review whether a practice has appropriate consequences for individuals and institutions, and especially to guard against adverse consequences (Messick, 1980). You . . . may prefer to exclude reflection on consequences from the meanings of the word validation, but you cannot deny the obligation.” (Cronbach, 1988, p. 6)
Lessons Learned:
Validation is never finished. Consequences may not immediately reveal themselves. Perseverance is required. Humility is an appropriate stance. As Jawaharlal Nehru advised:
“Let us be a little humble; let us think that the truth may not perhaps be entirely with us.”
Nehru made this statement on January 14, 1957, during a speech at the Indian Science Congress in Kolkata (then Calcutta). I have returned to this quote again and again in my work. It sets a very helpful stance for exploring complex topics.
Rad Resource
The Ninety-five Theses of Lee J. Cronbach and Associates that opened their 1980 classic text, Toward Reform of Program Evaluation. I return to these frequently to center myself in core assertions that have (almost all) stood the test of time.
Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.