Memorial Week In Evaluation: A pioneering study of evaluation use by Michael Quinn Patton

This is the opening post in a series commemorating pioneering evaluation publications in conjunction with Memorial Day in the USA (May 28). 

My name is Michael Quinn Patton, a former AEA President and recipient of AEA’s 2017 Research on Evaluation Award. In conjunction with Memorial Day, I am curating six AEA365 contributions featuring pioneering and classic evaluation publications.   I begin with the study that launched utilization-focused evaluation.

In 1975, as evaluation was emerging as a distinct field of professional practice, I undertook a study with colleagues and students of 20 federal health evaluations to assess how their findings had been used and to identify the factors that affected varying degrees of use. We interviewed the evaluators and those for whom the evaluations were conducted. That study marked the beginning of the formulation of utilization-focused evaluation (Patton, 1978) now in its 4^th edition (Patton, 2008).

 

 

 

 

 

 

 

 

 

In that original study, we asked respondents to comment on how, if at all, each of 11factors extracted from the literature on utilization had affected use of their evaluation. These factors were methodological quality, methodological appropriateness, timeliness, lateness of report, positive or negative findings, surprise of findings, central or peripheral program objectives evaluated, presence or absence of related studies, political factors, decision maker/evaluator inter-actions, and resources available for the study. Finally, we asked respondents to “pick out the single factor you feel had the greatest effect on how this study was used.”

From this long list of questions only two factors emerged as consistently important in explaining utilization: (1) political considerations, and (2) a factor we called the personal factor. This latter factor was unexpected, and its clear importance to our respondents had, we believed, substantial implications for the use of program evaluation. None of the other specific literature factors about which we asked questions emerged as important with any consistency. Moreover, when these specific factors were important in explaining the use or nonuse of a particular evaluation study, it was virtually always in the context of a larger set of circumstances and conditions related to either political considerations or the personal factor.

Lesson Learned:

The personal factor is the presence of an identifiable individual or group of people who personally care about the evaluation and the findings it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact. The personal factor represents the leadership, interest, enthusiasm, determination, commitment, assertiveness, and caring of specific, individual people. These are people who actively seek information to learn, make judgments, get better at what they do, and reduce decision uncertainties. They want to increase their ability to predict the outcomes of programmatic activity and thereby enhance their own discretion as decision makers, policymakers, consumers, program participants, funders, or whatever roles they play. These are the primary users of evaluation. (Patton, 2008, pp. 66-67)

The breakthrough in publication came when Carol Weiss published our findings as a chapter in her book on Using social research in public policy making (1977). That was the beginning of utilization-focused evaluation.

Rad Resources:

• Patton, M. Q., P. S. Grimes, K. M. Guthrie, N. J. Brennan, B. D. French, & D. A. Blyth.
• (1977). In Search of Impact: An Analysis of the Utilization of Federal Health Evaluation Research.” Pp. 141–64 in Carol H. Weiss, Using social research in public policy making,
• Lexington, MA: D. C. Heath. 
• Patton, M.Q. (1978). Utilization-Focused Evaluation. Beverly Hills, CA: Sage.
• Patton, M.Q. (2008). Utilization-Focused Evaluation.  4^th ed Thousand Oaks, CA: Sage.

The American Evaluation Association is celebrating Memorial Week in Evaluation. The contributions this week are remembrances of pioneering and classic evaluation publications. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.