Oregon Evaluators Week: Evaluating Through Simulations by Katie Street

Virtual greetings! Katie Street here, evaluator for Portland State University’s Child Welfare Training Unit Partnership (CWP) in Salem, Oregon.

One challenge faced by any training program is: How do we measure acquisition of trainee’s knowledge and skills? Other fields have used various forms of simulated workforce situations to answer this question for years, and recently the use of simulations has been gaining traction in the child welfare training field. At the CWP, we use standardized scenarios, trained actor “clients,” transformed office space, and individual video recording (as opposed to group recording) to set the stage for assessment and evaluation.  This provides trainees with a safe space to make mistakes and a sense of how they will perform on the job. Simulation videos are reviewed and assessed by the trainee, trainer, and supervisor. Currently, we incorporate five simulation scenarios into the training given to child welfare workers. They include parent and child interviews and court proceedings, direct and cross examination, and case presentation.

Hot Tip:

There are several variations of simulations you can choose from to fit the needs of your organization. Having actors or trainers portray clients produces a consistent experience which creates more standardized data that are easier to use in program evaluation. Alternatively, using students to role-play clients is cheaper, and can help develop empathy as students step into the shoes of a family involved in the child welfare system. Doing individual simulations also adds the opportunity for trainees to self-assess, which can solidify classroom learning. One trainee said, “I liked seeing myself, to be able to witness my own mistakes helped to solidify how to fix it in myself.”

Rad Resource:

Capacity Builders Center for the States has a handy cheat sheet that summarizes costs and benefits and the basic structure of incorporating simulation learning into your child welfare workforce training unit. Their handy recipe for a successful simulation includes:

  • Learning Objectives
  • Scenario/Environment
  • Participant and Actor Preparation
  • The Simulation Itself
  • Debriefing

It also includes some examples of states that have incorporated simulation training already, to get an idea of different approaches being utilized across the country.

Lessons Learned:

It’s great to have field experts grade simulations and give valuable feedback to participants. If your team has more than one grader, it’s important to meet regularly to make sure the criteria is clear and all graders are evaluating on the same standards. Providing and maintaining consistent expert feedback in this way can be resource and labor intensive but can also be a critical learning point for participants.

This week, AEA365 is featuring posts from evaluators in Oregon. Since Evaluation 2020 was moved from Portland, OR to online, a generous group of Oregon evaluators got together to offer content on a variety of topics relevant to evaluators. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.