ROE Bonus: Paul Brandon and Landry Fukunaga on Methods Used in Empirical Studies of Stakeholder Involvement in Program Evaluation

We are Paul Brandon and Landry Fukunaga from the University of Hawai‘i at M?noa. Stakeholder involvement in program evaluation is one of the most enduring topics in the program evaluation literature, but empirical research on the topic has been summarized only within limited domains. We conducted a literature review of empirical research, examining 7,580 publications from January 1985 through May 2010 that we identified in systematic searches of 11 major electronic databases. After reviewing abstracts of the publications, we closely examined 43 peer-reviewed articles that (a) described stakeholder involvement in the conduct of or the study of program evaluation and (b) collected data on stakeholder involvement. Our process eliminated reflective narratives and other reports that had did not discuss systematic data collection on involvement, articles about theory, book reviews, and literature reviews.

Lessons Learned: Of the 43 articles:

• 14 (32%) were about evaluation in general, 11 (26%) took place in the domains of education or health, 6 (14%) were about social services, and 1 (2%) was about environmental planning.
• 31 (72%) were about evaluations that collected data on stakeholder involvement in actual evaluations. Of these, 23 were single-case studies and 8 were multiple-case studies. The remaining 12 (28%) were research studies or simulations involving stakeholders that did not take place within an evaluation context.
• The types of stakeholder groups most frequently studied were program staff and/or implementers of the program (18, or 42%), program administrators or board members (16, or 37%), and evaluators (12, or 28%). An average of 2.13 of types of stakeholder groups was studied.
• 16 (37%) of the studies collected data on fewer than 25 stakeholder participants, 8 (19%) collected data on 26–100 participants, and 12 (28%) collected data on more than 100. The remaining 7 (16%) did not report the number of stakeholder participants or were simulations.
• The methods used to study stakeholder involvement included surveys in 28 (65%) of the studies, interviews in 27 (63%), document reviews in 12 (28%), observations in 11 (26%), personal reflections in 5 (12%), focus groups in 4 (9%), and the results of informal discussion in 3 (7%).

The studies paid very little attention to how the research was conducted.

We suggest that (a) the empirical literature on stakeholder involvement in program evaluation is less substantial than many might believe, (b) the quality of the literature in stakeholder involvement in program evaluation is impossible to analyze because of a lack of detail about research methods, and (c) the dearth of studies provides additional evidence for the claims that funding for research on evaluation is seriously lacking.

Hot Tip: For more detail regarding this study, check out the slides from our presentation at Evaluation 2010.

The American Evaluation Association is celebrated Research on Evaluation (ROE) Week with our colleagues in the ROE AEA Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.