Greetings fellow evaluators! I am Davina M. DeVries, Learning and Development Manager for the University of South Florida College of Pharmacy (COP). In my spare time I am a PhD student in the Measurement, Research, and Evaluation program, focusing on educational program evaluation.
The college of pharmacy asked me to develop and evaluate the system for tracking learning outcomes across the 4-year program by mapping exam questions to specific accreditation standards. This process required that I try several iterations of the tagging process to find the most efficient process for getting the system organized correctly. I would like to describe my approach to the Model for Collaborative Evaluation (MCE) in a rapid-fire evaluation. I chose to use the MCE to give stakeholders ownership of the process and use real-time feedback to make required corrections.
I asked the faculty how they currently use the mapping system, what corrections needed to be made, and how they would use the data. To aid in the process and act as subject matter experts, fourth year students, during their academic rotations, were selected to assist in the tagging process.
Professional programs are tasked with preparing student to excel in their field of choice, which is guarded by licensure exams. The COP uses a secure testing database to input exam questions and then tag questions to the various standards for our program. Faculty tend to overlook this step or avoid using the existing correct categories for tagging questions. Using the MCE I was able to identify key stakeholders and use open communication to find where the process had stopped working, and then used successive rapid-fire evaluations to determine a new process for updating the question database. With each iteration the approach students used to complete their task was updated.
- The MCE is a stakeholder involvement approach that allows for an open dialogue that supports determination of the project’s scope. Stakeholders can actively participate and have ownership of the outcomes to be able to use the outcomes in real time.
- The perceptions that the system needed to be 100% correct before it could be used again was incorrect. This process is a gradual change that improves over time.
- Faculty are busy, so recruiting students to assist helps the faculty and the student. 4th year students are studying for the licensure exam and get a boost by working on this project.
- Allow those assisting to find their own way of completing the required tasks. No two students want to review questions in the same manner.
- Choose one course at a time and work diligently to review all questions.
- Train faculty to take charge once their course’s questions are completely tagged. The faculty must have ownership of the database.
- Collaborative Evaluations Step-by-Step https://www.sup.org/books/title/?id=21594
- A Video Introduction to the Model for Collaborative Evaluation https://www.youtube.com/watch?v=aZBriebheZQ
The American Evaluation Association is celebrating Health Professions Education Evaluation and Research (HPEER) TIG Week with our colleagues in the Health Professions Education Evaluation and Research Topical Interest Group. The contributions all this week to aea365 come from our HPEER TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.