Hi fellow evaluators. I am Sebastian Lemire of Abt Associates, and I am kicking off the Research on Evaluation (RoE) AEA365 Blog Week on behalf of the RoE TIG Leadership. This week’s blogs will showcase current research on the methods, contexts, activities, effects, and best practices within evaluation. Each blog highlights implications for how we understand our work as evaluators, and thus offers insights to strengthen our collective practice.
As a scholar-practitioner, I have designed and implemented a broad range of theory-based evaluations over the past 15 years. Informed by my own practice, I have become increasingly interested in how we use different types of theory in evaluation. More specifically, my interest centers on two questions: How do we develop program theories? And how, if at all, do we make use of social science theory in developing these?
Answering these questions is particularly salient in the context of theory-based evaluation, where the central role of developing and testing a program theory provides particularly fertile grounds for integrating social science theory. As part of a recent systematic review of 131 published theory-based evaluations, Tina Christie, Stewart Donaldson, Alexander Kwako, and I are now able to shed some empirical light on these questions.
Lessons Learned: How do evaluators develop program theories?
Based on our review, almost half (49%) of the program theories were developed in a participatory fashion (involving program staff, program funders, or other stakeholders), while one-third (37%) were developed in non-participatory ways (e.g., the evaluator developing the program theory). In terms of sources, program theories are commonly grounded on some combination of stakeholders, existing literature, and program documents (43%); though, some rely solely on existing literature (18%) or stakeholders (12%). The sources are unspecified for a sizeable proportion of program theories (19%).
Lessons Learned: Do evaluators make use of social science theory in developing program theories?
As shown in Figure 1., most published examples of theory-based evaluations (60%) do not involve explicit use of social science theory. Some evaluators (17%) reference social science theory without specifying how and in what way the social science theory was used in developing the program theory—we refer to this as “theory labeling”. Finally, and of particular interest, we also found that one-fourth of the evaluators (23%) engage in theory knitting, using social science theory as the conceptual grounding for developing the program theory. We refer to this as “theory knitting”: the explicit and purposeful use of social science theory in developing a program theory.
Rad Resources: Curious about theory knitting?
Consider the following resources:
Riemer, M., & Bickman, L. (2011). Using program theory to link social psychology and program evaluation. In M. M. Mark, S. I. Donaldson, and B. Campbell (Eds.), Social psychology and evaluation. New York, NY: Guilford Press.
Riggin, L. J. C. (1990). Linking program theory and social science theory. In L. Bickman (Ed.), Advances in program theory. New Directions for Program Evaluation, 47, 109-120. San Francisco, CA: Jossey-Bass.
The American Evaluation Association is celebrating Research on Evaluation (RoE) TIG Week with our colleagues in the RoE Topical Interest Group. The contributions all this week to aea365 come from our RoE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.
Hello Sebastian,
This is great information to know! It is interesting how one-third of evaluators developed the theory as they worked along in the program. It comes to show that there is always room to learn in unexpected ways.
Interesting – thank you for sharing this!
There is nothing wrong with theory knitting. The theory resulting from any two (or more) theories, rigorously knitted, will almost certainly be better than any one of the “source” theories (and, the whole is more than the sum of the parts.
We use an almost identical method for integrating/synthesizing theories. Here, as an example, you can see multiple propositions (which might be seen as being from different theories) knitted/integrated together http://meaningfulevidence.com/wp-content/uploads/Basics-of-IPA.pdf The process is really so simple. The only thing I would add is the use of IPA to evaluate the structure of the theories. This allows you to see how much progress you have made in your knitting… and to see how much farther you need to go.