Hi fellow evaluators. I am Sebastian Lemire of Abt Associates, and I am kicking off the Research on Evaluation (RoE) AEA365 Blog Week on behalf of the RoE TIG Leadership. This week’s blogs will showcase current research on the methods, contexts, activities, effects, and best practices within evaluation. Each blog highlights implications for how we understand our work as evaluators, and thus offers insights to strengthen our collective practice.
As a scholar-practitioner, I have designed and implemented a broad range of theory-based evaluations over the past 15 years. Informed by my own practice, I have become increasingly interested in how we use different types of theory in evaluation. More specifically, my interest centers on two questions: How do we develop program theories? And how, if at all, do we make use of social science theory in developing these?
Answering these questions is particularly salient in the context of theory-based evaluation, where the central role of developing and testing a program theory provides particularly fertile grounds for integrating social science theory. As part of a recent systematic review of 131 published theory-based evaluations, Tina Christie, Stewart Donaldson, Alexander Kwako, and I are now able to shed some empirical light on these questions.
Lessons Learned: How do evaluators develop program theories?
Based on our review, almost half (49%) of the program theories were developed in a participatory fashion (involving program staff, program funders, or other stakeholders), while one-third (37%) were developed in non-participatory ways (e.g., the evaluator developing the program theory). In terms of sources, program theories are commonly grounded on some combination of stakeholders, existing literature, and program documents (43%); though, some rely solely on existing literature (18%) or stakeholders (12%). The sources are unspecified for a sizeable proportion of program theories (19%).
Lessons Learned: Do evaluators make use of social science theory in developing program theories?
As shown in Figure 1., most published examples of theory-based evaluations (60%) do not involve explicit use of social science theory. Some evaluators (17%) reference social science theory without specifying how and in what way the social science theory was used in developing the program theory—we refer to this as “theory labeling”. Finally, and of particular interest, we also found that one-fourth of the evaluators (23%) engage in theory knitting, using social science theory as the conceptual grounding for developing the program theory. We refer to this as “theory knitting”: the explicit and purposeful use of social science theory in developing a program theory.
Rad Resources: Curious about theory knitting?
Consider the following resources:
Riemer, M., & Bickman, L. (2011). Using program theory to link social psychology and program evaluation. In M. M. Mark, S. I. Donaldson, and B. Campbell (Eds.), Social psychology and evaluation. New York, NY: Guilford Press.
Riggin, L. J. C. (1990). Linking program theory and social science theory. In L. Bickman (Ed.), Advances in program theory. New Directions for Program Evaluation, 47, 109-120. San Francisco, CA: Jossey-Bass.
The American Evaluation Association is celebrating Research on Evaluation (RoE) TIG Week with our colleagues in the RoE Topical Interest Group. The contributions all this week to aea365 come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com.