We are from the College of Education and Human Sciences, University of Nebraska-Lincoln, with Timothy Guetterman, the doctoral student, and Delwyn Harnisch, the Professor.
Mixed methods approaches can be useful in assessing needs and readiness to learn among professional workshop participants. Combining qualitative and quantitative methods can enhance triangulation and completeness of findings. We recently mixed methods while evaluating a weeklong workshop delivered to medical educators in Kazakhstan and experienced how mixing can aid evaluation activities.
The international collaboration between teams in the U.S. and Kazakhstan presented challenges that we mitigated through technologies, such as email, Skype, and Dropbox. Surveys administered before, during, and after the workshop through an online tool, Qualtrics, were important to guide implementation, continually assess learning, and understand the participant’ perspective.
Hot Tips:
- Guiding Implementation. Mixed methods within the needs and readiness assessment served a formative purpose, helping us tailor the workshop to specific participant needs. Mixed methods analyses yielded rich details about what participants wanted and needed that would be difficult to anticipate with a quantitative instrument. Online surveys presented a way to connect with participants early. Beyond quantitative scales, we asked questions (e.g., “What do you hope to learn?”). Because data were immediately available, findings guided the workshop implementation.
- Continually Assess Learning. Throughout the workshop, brief (about one minute) surveys at the end of the day helped to gauge understanding of where participants are to develop the community of learners. Providing a daily survey solicited brief qualitative responses from items (e.g., “Summarize in a few words the most important point from today”; “What point is still confusing?”). The questions provided valuable information but only took minutes to complete.
- Understand the Participants’ Perspectives. In the summative evaluation of the workshop, mixed methods allowed us to obtain participant ratings and gain understanding of what participants learned through open-ended qualitative questions.
Lessons Learned:
- With the use of these tools, we were able to model in this workshop a process for developing a deep and practical understanding of assessment for learning. With the leaders at this program sharing at their sites, we are beginning to see the vehicle of site-based teacher learning communities. Each of these sites is using two or three techniques in their own classrooms and then meeting with other colleagues monthly to discuss their experiences and to see what other teachers are doing.
- The result of this effort is that these teacher learning communities now develop a shared language enabling them to talk to one another about what they are doing.
- In short, the use of mixed methods allows the team to focus on where the learners are now, where they want to go, and how we can help them get there.
The American Evaluation Association is celebrating the Business, Leadership, and Performance TIG (BLP) Week. The contributions all week come from BLP members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.