Virtual greetings! Katie Street here, evaluator for Portland State University’s Child Welfare Training Unit Partnership (CWP) in Salem, Oregon.
One challenge faced by any training program is: How do we measure acquisition of trainee’s knowledge and skills? Other fields have used various forms of simulated workforce situations to answer this question for years, and recently the use of simulations has been gaining traction in the child welfare training field. At the CWP, we use standardized scenarios, trained actor “clients,” transformed office space, and individual video recording (as opposed to group recording) to set the stage for assessment and evaluation. This provides trainees with a safe space to make mistakes and a sense of how they will perform on the job. Simulation videos are reviewed and assessed by the trainee, trainer, and supervisor. Currently, we incorporate five simulation scenarios into the training given to child welfare workers. They include parent and child interviews and court proceedings, direct and cross examination, and case presentation.
Hot Tip:
There are several variations of simulations you can choose from to fit the needs of your organization. Having actors or trainers portray clients produces a consistent experience which creates more standardized data that are easier to use in program evaluation. Alternatively, using students to role-play clients is cheaper, and can help develop empathy as students step into the shoes of a family involved in the child welfare system. Doing individual simulations also adds the opportunity for trainees to self-assess, which can solidify classroom learning. One trainee said, “I liked seeing myself, to be able to witness my own mistakes helped to solidify how to fix it in myself.”
Rad Resource:
Capacity Builders Center for the States has a handy cheat sheet that summarizes costs and benefits and the basic structure of incorporating simulation learning into your child welfare workforce training unit. Their handy recipe for a successful simulation includes:
- Learning Objectives
- Scenario/Environment
- Participant and Actor Preparation
- The Simulation Itself
- Debriefing
It also includes some examples of states that have incorporated simulation training already, to get an idea of different approaches being utilized across the country.
Lessons Learned:
It’s great to have field experts grade simulations and give valuable feedback to participants. If your team has more than one grader, it’s important to meet regularly to make sure the criteria is clear and all graders are evaluating on the same standards. Providing and maintaining consistent expert feedback in this way can be resource and labor intensive but can also be a critical learning point for participants.
This week, AEA365 is featuring posts from evaluators in Oregon. Since Evaluation 2020 was moved from Portland, OR to online, a generous group of Oregon evaluators got together to offer content on a variety of topics relevant to evaluators. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Katie,
First off, thank you for your post and your hot tip/rad resources, they are very useful!
I, like many other commentors here, am in the process of completing my Professional Masters of Education (PME) at Queens University. I came across your article when searching through AEA365 for simulation and evaluation articles. I was quite happy to read yours as a lot of what you wrote about coincides with some of my work at Humber College as an instructor for the Paramedic Program. Simulation training is huge in our program and many other vocational learning programs as well. We use this specifically to train and enhance our students’ technical skills for real-world application as a Paramedic. I was quite glad to see how other programs are utilizing simulation training and I hope the results are positive in all aspects! We are currently working towards using Augmented and Virtual Reality (AR/VR) training to further improve our simulation training and I think this could be very good for our students!
The lesson you learned about having field experts is something we also value when running our simulations. We always try to have field experts be the evaluators for the scenarios and that way, the students can also get authentic and applicable feedback to enhance their practice.
I wanted to share one of our assessment/evaluation tools that we use for our simulations. I believe the definitions can be changed and adapted to specific needs and learning objectives for specific programs. ***Please refer to pages 4-5 of the PDF***
https://www.rppeo.ca/component/rsfiles/download-file/files.html?path=Forms%255CEducation%255CSelf-Assessment%2BForms%255CCompetency%2BAssessment%2B-%2BJuly%2B2013.pdf&Itemid=4487
Thanks again for your post!
Hi Katie, I am currently enrolled in a Master of Education Program at Queens University and am just wrapping up a course on Program Inquiry and Evaluation. I am a Training and Development Coordinator in Residence Life at McMaster University in Ontario, Canada. It has been an eye-opening program for me in terms of pre-planned evaluations and incorporating that long term in to training planning. When it comes to case based, or scenario type training it has been my experience that the feedback and debrief conversations are so important to ensure participants understand what approach they should take or where they need more practice. Have you considered at all pre-planning this into an overall evaluation plan? Also, with COVID-19 playing such in important role in today’s world- how have you adjusted your approach to this type of training? We are currently offering multiple Asynchronous and Synchronous case scenario’s in to our training plan, and I was wondering if you’ve had any recent successes with implementation but also evaluation of those in a digital world?
Thank you,
Tara Roberts
Hi Katie!
I found your idea to use simulations for evaluation both intriguing and practical. I am currently completing my Masters of Education through Queens University, and was thinking about how simulations could be incorporated in the evaluation of an education-related program.
The text by Angela E. Horton called “Nurse Educator Perceptions of Using Simulation for Evaluation of Nursing Competencies” describes the use of simulation in nursing to address mistakes, it explains that “[t]here are times when nurses’ critical thinking and judgment may be in question, such as when a medication error has occurred, or there is a lapse in judgment. The education department is then responsible for conducting an assessment to determine if a nurse’s practice is safe.” (2020). Similarly in the field of education, teachers, students and administrators have an abundance of guidelines, rules and regulations to follow. When a mistake is made, simulation could be a great way to incorporate critical thinking into reenacting the situation and thoroughly processing the right way to approach the situation the next time.
In regard to evaluation in education, I think simulation could also be useful. Take a reading program for example. If one is evaluating this reading program, the evaluator can use simulation to show teachers new methodologies that are being introduced and can also simulate the current program to diagnose any current problems.
I also enjoyed reading your “rad resource”. I thought the steps were very clear and would be easy to follow for anyone using simulation in child welfare training, but also other areas of evaluation as well. Queues such as “encourag[ing] participants to stop when they feel stuck and ask for assistance” and “[i]f the goal is evaluation, clarify that participants must finish the simulation before receiving feedback” would be useful to have while conducting one’s first evaluation through simulation.
Thanks so much for your post! It is a pleasure to be introduced to new ways of approaching evaluation.
Jen
References
Horton, A. E. Nurse Educator Perceptions of Using Simulation for Evaluation of Nursing Competencies, ProQuest Dissertations Publishing, 2020.
Hello Sheila and Katie,
Thank you for the perspective and tips on using simulations as a form of evaluation. I am currently studying Program Inquiry and Evaluation through Queen’s University (Master’s Program). I noticed that in your method, each trainee has a recorded video for them to review their mistakes and see what they did well. As part of the Program Evaluation Standards – feasibility standards, it is important for evaluation methods to be cost effective. I would imagine that this method is very helpful and thorough, however, what type of corporation would be able to invest into such a costly evaluation? What type of programs do you feel must have such a thorough evaluation?
As in most evaluation scenarios, it’s difficult to maintain a balance between cost and quality. It was intriguing to read about how comprehensive an evaluation could be, bringing in all stakeholders and directly providing useful feedback to the trainees.