This is the opening post in a series commemorating pioneering evaluation publications in conjunction with Memorial Day in the USA (May 28).
My name is Michael Quinn Patton, a former AEA President and recipient of AEA’s 2017 Research on Evaluation Award. In conjunction with Memorial Day, I am curating six AEA365 contributions featuring pioneering and classic evaluation publications. I begin with the study that launched utilization-focused evaluation.
In 1975, as evaluation was emerging as a distinct field of professional practice, I undertook a study with colleagues and students of 20 federal health evaluations to assess how their findings had been used and to identify the factors that affected varying degrees of use. We interviewed the evaluators and those for whom the evaluations were conducted. That study marked the beginning of the formulation of utilization-focused evaluation (Patton, 1978) now in its 4th edition (Patton, 2008).
![1st edition of Utilization-Focused Evaluation, 1978](http://aea365.org/blog/wp-content/uploads/2018/05/UFE-78.png)
![4th edition of Utilization-Focused Evaluation, 2008](http://aea365.org/blog/wp-content/uploads/2018/05/UFE-08.png)
In that original study, we asked respondents to comment on how, if at all, each of 11factors extracted from the literature on utilization had affected use of their evaluation. These factors were methodological quality, methodological appropriateness, timeliness, lateness of report, positive or negative findings, surprise of findings, central or peripheral program objectives evaluated, presence or absence of related studies, political factors, decision maker/evaluator inter-actions, and resources available for the study. Finally, we asked respondents to “pick out the single factor you feel had the greatest effect on how this study was used.”
From this long list of questions only two factors emerged as consistently important in explaining utilization: (1) political considerations, and (2) a factor we called the personal factor. This latter factor was unexpected, and its clear importance to our respondents had, we believed, substantial implications for the use of program evaluation. None of the other specific literature factors about which we asked questions emerged as important with any consistency. Moreover, when these specific factors were important in explaining the use or nonuse of a particular evaluation study, it was virtually always in the context of a larger set of circumstances and conditions related to either political considerations or the personal factor.
Lesson Learned:
The personal factor is the presence of an identifiable individual or group of people who personally care about the evaluation and the findings it generates. Where such a person or group was present, evaluations were used; where the personal factor was absent, there was a correspondingly marked absence of evaluation impact. The personal factor represents the leadership, interest, enthusiasm, determination, commitment, assertiveness, and caring of specific, individual people. These are people who actively seek information to learn, make judgments, get better at what they do, and reduce decision uncertainties. They want to increase their ability to predict the outcomes of programmatic activity and thereby enhance their own discretion as decision makers, policymakers, consumers, program participants, funders, or whatever roles they play. These are the primary users of evaluation. (Patton, 2008, pp. 66-67)
The breakthrough in publication came when Carol Weiss published our findings as a chapter in her book on Using social research in public policy making (1977). That was the beginning of utilization-focused evaluation.
Rad Resources:
- Patton, M. Q., P. S. Grimes, K. M. Guthrie, N. J. Brennan, B. D. French, & D. A. Blyth.
- (1977). In Search of Impact: An Analysis of the Utilization of Federal Health Evaluation Research.” Pp. 141–64 in Carol H. Weiss, Using social research in public policy making,
- Lexington, MA: D. C. Heath.
- Patton, M.Q. (1978). Utilization-Focused Evaluation. Beverly Hills, CA: Sage.
- Patton, M.Q. (2008). Utilization-Focused Evaluation. 4th ed Thousand Oaks, CA: Sage.
The American Evaluation Association is celebrating Memorial Week in Evaluation. The contributions this week are remembrances of pioneering and classic evaluation publications. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Michael,
First off, I want to introduce myself. My name is Matthew Walton and I am a graduate student at Canada’s Queens University. As part of one of my courses, I have been tasked with locating an AEA365 post that interests me and commenting on it. One of the concepts that I have found intriguing is the idea of evaluation use. That interest led me to this blog post.
The research findings surrounding the personal factor discussed in this post stimulated curiosity about what an evaluator can do to promote evaluation buy-in from and individual or group. This train of thought led me to an article written by Taut (2007). Here the author states that involving stakeholders in the evaluation process results in changes in participant feelings towards the findings. Connecting these two ideas seems logical to me. If program personnel is involved in the evaluation then they are more likely to care about the findings. If an individual or group of individuals care about the findings then they are more likely to utilize them. This connection of ideas has solidified the importance that I place on stakeholder involvement.
Thank you for your work on this topic.
Matt
References
Taut, S. (2007). Methodological and conceptual challenges in studying evaluation process use. Canadian Journal of Program Evaluation, 22(2), 1-20.