RoE TIG Week: Institutionalizing Evaluation in the U.S. Federal Government by, Esther C. Nolton, PhD

Greetings! I am Esther Nolton and I recently completed my PhD in Research and Evaluation Methods at George Mason University. As a GEDI Scholar, I had an opportunity to witness large-scale changes in the culture of valuing and capacity for using evaluation in the U.S. Federal government when the Evidence Act (P.L. 115-435) was enacted. Although this legislation is a leap forward for evaluation, it was unclear which levers drove institutional change toward increased prioritization of evaluation over time. This inspired me to investigate the process by which evaluation has been institutionalized in the U.S. Federal government and identify the factors that have promoted or impeded that process.

Rad Resource: Using Ethnography to Map Processes

Mapping the processes and mechanisms (in)effectively established to influence value in evaluation can inform more efficient decision-making processes. I make a small distinction between the value of evaluation (i.e., a statement of worth) and value in evaluation (i.e., an indication of belief and investment in that worth). This may be demonstrated by investing resources toward greater capacity to do and use evaluation. Each fiscal year, the U.S. President’s Budget appends rationale and analyses in the Analytical Perspectives. I adapted an ethnographic mapping method—combining interviews with 15 evaluation thought leaders with content and discourse analyses of the Analytical Perspectives from the last 25 years—to capture shifts in the prioritization of evaluation over time.

Lessons Learned: The Process is Cumulative and Cyclical

It was discovered that the Analytical Perspectives have codified system-wide rhetoric to portray the distinctiveness and legitimacy of evaluation. The process is a cumulative progression from cognitive, motivational, behavioral, to structural influences. This means micro-level actions have led to changes in macro-level structures and early stages of influence must be established to sustain widespread institutional change. It was also revealed that negative influences at any stage of the process could undo prior progress. The following figure illustrates the identified process.

Hot Tip: A Healthy Culture of Evaluation is the Highest Priority

Documents and lived experiences reported by informants revealed a number of actors, events, and policies that have influenced the (de)institutionalization of evaluation, which continue to impact current policymaking processes. Additionally, many efforts were modeled from strategies employed by trailblazing agencies and related disciplines (i.e., mimetic isomorphism), but adoption has been non-linear and disproportionate throughout the Federal government. Influences manifest in different ways, ranging from a belief system of value and trust in evaluation to a dynamic infrastructure to do and use more evaluation (see figure below). Therefore, cultivating a culture of readiness; fostering advocates; equipping evaluators and evaluation units; and establishing a robust evidence infrastructure are essential to institutionalizing evaluation.

Lesson Learned: Although value in evaluation is increasingly ubiquitous in the U.S. Federal government, recent behavioral and structural influences will be futile if evaluations are merely done for compliance without any genuine interest for utilization. These findings may be instrumental to policymakers and organizational leaders who continue to work toward institutionalizing evaluation in the Federal government or other complex organizations.

References:

Haveman, H. A. (1993). Follow the leader: Mimetic isomorphism and entry into new markets. Administrative Science Quarterly, 38(4), 593-627.

Lemire, S., Fierro, L. A., Kinarsky, A. R., Fujita-Conrads, E., & Christie, C. A. (2018). The U.S. federal evaluation market. In S. B. Nielsen, S. Lemire, & C. A. Christie (Eds.), The evaluation marketplace: Exploring the evaluation industry. New Directions for Evaluation, 160, 63-80. https://doi.org/10.1002/ev.20343

Li, Y. (2017). A semiotic theory of institutionalization. Academy of Management Review, 42(3), 520-547. https://doi.org/10.5465/amr.2014.0274

Wedel, J. R. (2019). The mapping method: A guide to charting corruption and influence processes [Working paper forthcoming]. Global Integrity.

---

The American Evaluation Association is celebrating RoE TIG Week with our colleagues in the Research on Evaluation TIG. All of the blog contributions this week come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.