Conducting Evaluation in Academic Medical Centers by Anne Vo and Jacob Schreiber

Hi there! We’re Anne Vo, Ph.D., Director of the Keck Evaluation, Institutional Reporting, and Assessment (KEIRA) Office at the Keck School of Medicine of USC, and Jacob Schreiber, Evaluation Assistant at KEIRA. Today, we offer reflections on what we’ve learned about conducting evaluation within an academic medical center—an environment that offers rich opportunities to observe, conduct, and understand evaluation practice and policy.

Hot Tip #1: Standards Rule Healthcare, Medicine, and Medical Education

Medicine is a highly regulated field. Broad swaths of stakeholders—clinicians, clinical educators, school and hospital administrators—rely on standards to inform decision-making and drive practice. As such, systematic evaluation often manifests as high turn-around monitoring of easily quantifiable outcomes (e.g., student academic performance, residency program match rates, etc.). Successfully “chasing numbers” enables organizations such as academic medical centers to communicate that standards of care and teaching are being met. Because standards offer a common language that stakeholders can use to think through pressing issues of the day, they also become the go-to frame of reference for decision-makers throughout the organization.

Rad Resource:

• Liaison Committee on Medical Education’s Standards for Medical School Accreditation
• Accreditation Council for Graduate Medical Education Competencies

Hot Tip #2: Everything is “Evaluated,” Everyone is an “Evaluator”

Because standards drive practice in Medicine, evaluation could become a decentralized activity. Aspects of evaluative practice—from question formulation, to data collection, monitoring, analysis, and synthesis—can often be divided among various stakeholder groups across an organization. This cascaded evaluation model emphasizes “local expertise” and echoes “team values” to which healthcare teams aspire. It is reminiscent of development evaluations that organizations such as UNICEF and the UNDP strongly support. And, in the medical context, it is a model that tends to immerse clinical experts in monitoring processes and largely distances them from actual evaluation.

Rad Resource:

• World Food Programme’s Decentralized Evaluation Quality Assurance System
• Models and approaches to decentralized evaluation system presented by Keith Child to the Consultative Group for International Agricultural Research

Hot Tip #3: Democratic Decision Making is a Core Value

Decisions about how medical education is done are often made through committees and guided by accreditation standards; specifically, LCME Standard 1 on Mission, Planning, Organization, and Integrity (see above link) and Standard 2 on Leadership and Administration. Academic and administrative committees oversee and monitor the quality of Undergraduate Medical Education (what we know as the first four years of medical school). Many of the same stakeholders serve across committees as well as the sub-committees and work groups within each. For evaluation to be meaningful, expect to have many of the same conversations with the same people on different levels. Most importantly, know what each committee’s charge is; its membership; and members’ roles and stances on issues that are up for discussion.

Rad Resource:

Alkin, M.C. and Vo, A.T. (2017). What Is the Organizational, Community, and Political Context of the Program? (pp. 77-87). In Evaluation Essentials: From A to Z (2^nd Edition). New York, NY: Guilford Press.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.