I’m Bernadette Wright, partner at Meaningful Evidence, LLC. In this post, I’m going to tell you about a new method for integrating your background research, to strengthen your evaluation results.
Lessons Learned:
“Little Is Known”?
I’ve seen that many program evaluation reports include short “background research” sections that quickly decide “little is known.” At the same time, a myriad of evaluations and related studies have created a vast and growing storehouse of information that is often overlooked. Moat and colleagues, for example, in their paper “Twelve myths about systematic reviews for health system policymaking,” counted 1,736 systematic reviews of studies about strategies and approaches for health care systems in one journal alone.
The vast volume of disconnected studies makes it hard for evaluators and practitioners to find and select the most effective ideas and theories to apply. Managers and researchers may start from scratch with a new theory. Or they may use a well-known theory that may not be the most relevant or effective theory.
Integrating Theories: The Missing Leg
Developed by Meaningful Evidence’s Dr. Steve Wallis, Integrative Propositional Analysis (IPA) is a method to rigorously assess, visualize, and integrate theories across studies. Unlike other methods, IPA lets us quantitatively assess theories based on their structures. As Cotae noted in a 2015 paper in Ecoforum, IPA “ensures a broad understanding of the challenges,” helping decision-makers to “reflect on the opportunities for implementation and chances of success.” The idea is, we need both credible data and well-structured theories to get more meaningful and useful evaluation results. In practice, involving stakeholders is the third leg of this approach.
Ideas for Evaluators
For your next evaluation, consider using IPA to diagram your theory based on your background research. Later, you might incorporate knowledge gained from your evaluation into your original theory and diagram. This would support other researchers to include your theory in meta-theoretical analyses. It would also support managers and funders to better understand and assess their options, so they can develop more effective solutions and accelerate their success.
Rad Resources:
- IPA white papers – These three white papers explain the IPA method and how to use it to assess, integrate, and improve theories and strategic plans.
- BetterEvaluation, “Synthesise Data Across Evaluations” – This webpage summarizes various methods for synthesizing data across evaluations and provides useful links to more information on each method.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.