Hello, my name is Tammy Calise, Co-Lead for Evaluation & Research at JSI Research & Training Institute, Inc. (JSI) and Instructor of Evaluating Today’s Complex Public Health a required course for the Master in Public Health Generalist program at the Harvard T.H. Chan School of Public Health. These roles allow me to raise awareness and foster an understanding of the value of implementation evaluation among professionals worldwide.
As a public health professional, working in the field for over 25 years, I recognize the significance of leveraging evidence to enhance access to a broader array of high-quality interventions, increase workforce productivity, and optimize public and private resources. As a self-identified evaluator, however, I have seen that the value of the end-user may vary, and findings from even the most rigorous research and evaluations may fail to have a significant impact if they are not contextually appropriate.
Complexity is undeniable in today’s landscape—be it in public health, education, business, or any other sector. When instructing students on evaluation methodologies, I emphasize the importance of context concerning evidence, the significance of describing relevant factors where an intervention is taking place, providing background to enhance the understanding and interpretation of data, and ensuring the applicability of findings is understood. Let me provide some context…
Over the years, I have found that most of my students recognize that randomized controlled trials have been identified as the “most rigorous” in terms of establishing evidence. They are also mostly familiar with quasi and other experimental designs. However, what is less understood is why these traditionally preferred approaches are not suitable for evaluating complex, adaptive, and innovative interventions nor do they always provide the type of evidence that can be used to make judgments or inform decisions.
The success of complex interventions, particularly in reaching their target populations, heavily depends on how they are implemented within a specific setting or situation. Thinking like this from an evaluative perspective is, in and of itself, a social change. As a catalyst for change, I emphasize the importance of implementation evaluations. It is my goal to ensure students appreciate that these evaluations generate the type of evidence essential for practical application in the real world because they offer insights into the structures, resources, and processes that facilitate delivery; the quantity and quality of services provided; how intervention activities, and participants’ interactions with them, cause change; and what (and how) external factors influence the delivery and functioning of interventions.
To demonstrate these concepts and provide my students with an opportunity to apply new knowledge and skills, I have them develop an evaluation plan for a hypothetical intervention. I provide them with a scenario, including multi-level activities, anticipated outputs, and outcomes. Students are asked to think about the theory of change and develop a logic model. I purposely give them minimal information and tell them to ask me, as the implementation team, any information they need to understand the intervention and develop the plan. Inevitably, students suggest randomization, propose a comparison group, or consider some other experimental design, which is not realistic or feasible given the complex initiatives they are evaluating. To demonstrate real life and the importance of context, I cut their budget, modify the intervention, and inform them of a new policy implemented outside their control that will impact implementation and evaluation. Students develop a deeper understanding of the field and recognize the importance of a comprehensive definition of evidence while navigating these challenges, one that implementation evaluations can provide.
Rad Resources
The American Evaluation Association is hosting Teaching of Evaluation TIG week. All posts this week are contributed by members of the ToE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.