Hello! I’m Andrew Nelson, a medical student at the Keck School of Medicine of USC and Lead Student Director of the Keck Online Learning Initiative (KOLI). We are a student-run organization dedicated to the creation of curriculum-based, online supplemental study material for medical students. When my colleagues started KOLI in 2015, they wanted to build an organization that was designed not only to last, but to improve. Doing so involved invoking methods of program design and evaluative thinking that, despite our collective years in higher education, were mysteries to us. However, thanks to outstanding support and advising from our mentor, Dr. Anne Vo, we have developed a model for a student-run peer education organization that we believe will support continuous learning and improvement.
Hot Tip #1: Create a logic model in which each stakeholder has a clearly defined role.
It is important in any organization for every member to know their responsibilities. This is especially true when the members are busy medical students and faculty members. In KOLI, member roles change every year as students matriculate and graduate. As such, it is crucial for the longevity of the organization that new leaders understand the roles that they are undertaking.
Rad Resources:
We used a combination of resources to facilitate discussions on these topics. The chapter on “How Do You ‘Understand’ the Program?” in Evaluation Essentials: From A to Z (2nd Edition) offered accessible and insightful tips for creating a logic model. Kara Crohn’s work on process mapping has also been influential.
Hot Tip #2: Foster positive relationships with faculty and administration.
Our organization would not exist without the administration’s buy-in, and we would not have content without our faculty educators. By demonstrating the utility and popularity of our study tool to the administration and sharing our program design in various administrative and academic venues, we have received increased resources and faculty support. Developing relationships with faculty has allowed us to tailor our content towards the most important information.
Rad Resources:
Curriculum Committee Meetings and 1-on-1 meetings with faculty and administration have been very productive and constructive spaces for us to share our work. So, too, have local and regional medical education conferences (e.g., Innovations in Medical Education Conference, AAMC’s Western Group on Educational Affairs Meeting).
Hot Tip #3: Reflect – evaluate and re-evaluate the outcomes you want to measure.
Because we provide a curriculum-based resource, our initial outcome of interest was users’ curriculum exam performance. However, in our discussions at AEA 2018, we were advised to also consider users’ long-term outcomes, such as development of self-directed learning skills later in their medical education. The Institute for Healthcare Improvement’s work on the PDSA cycle has been a cornerstone resource for us in the program design and implementation phases. Thanks to this kind of thinking, we anticipate that it will continue to be a helpful resource as we move forward on this charge.
The American Evaluation Association is celebrating Health Professions Education Evaluation and Research (HPEER) TIG Week with our colleagues in the Health Professions Education Evaluation and Research Topical Interest Group. The contributions all this week to aea365 come from our HPEER TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hello Andrew,
Thank you for a great article. I am currently enrolled in a professional M.Ed. to further my knowledge as a Continuing health sciences education program coordinator and found your article helped me apply my current learning.
The description of your program lends itself well to the logic model or theory of change visualization of a program’s resources, activities and expected outcomes. As you mentioned, curriculum exam performance provides a measurable short-term outcome, while the development of self-directed learning skills later in their medical education provides a measurable long-term outcome. Applying the logic model to define stakeholder roles is an especially useful idea.
I will certainly be implementing Plan-Do-Study-Act (PDSA) tool as a test of change in my organization. The PDSA resource you shared led me to the Health Care Data Guide by Lloyd P. Provost & Sandra K. Murray (2011) – which provides tools and strategies to use data for the improvement of health care processes and systems.
You mentioned that KOLI would not exist without the administration’s buy-in and that by sharing your program design in various administrative and academic venues you have increased resources and faculty support. When conducting curriculum and program evaluations for KOLI, how do you maintain the integrity by not consciously or unconsciously guiding results to act in a favourable way to support your efforts in attracting resources and faculty support? I only ask because as part of my studies we recently reviewed the Commitment to Standards of Practice from the Joint Committee on Standards for Educational Evaluation which caused me to reflect on my own evaluation efforts.
Regards,
Shawn
Hello Anne,
Thank you for a great article. I am currently enrolled in a professional M.Ed. to further my knowledge as a Continuing health sciences education program coordinator and found your article helped me apply my current learning.
The description of your program lends itself well to the logic model or theory of change visualization of a program’s resources, activities and expected outcomes. As you mentioned, curriculum exam performance provides a measurable short-term outcome, while the development of self-directed learning skills later in their medical education provides a measurable long-term outcome. Applying the logic model to define stakeholder roles is an especially useful idea.
I will certainly be implementing Plan-Do-Study-Act (PDSA) tool as a test of change in my organization. The PDSA resource you shared led me to the Health Care Data Guide by Lloyd P. Provost & Sandra K. Murray (2011) – which provides tools and strategies to use data for the improvement of health care processes and systems.
You mentioned that KOLI would not exist without the administration’s buy-in and that by sharing your program design in various administrative and academic venues you have increased resources and faculty support. When conducting curriculum and program evaluations for KOLI, how do you maintain integrity by not consciously or unconsciously guiding results to act in a favourable way to support your efforts in attracting resources and faculty support? I only ask because as part of my studies we recently reviewed the Commitment to Standards of Practice from the Joint Committee on Standards for Educational Evaluation which caused me to reflect on my own evaluation efforts.
Regards,
Shawn