Hi, Jennifer Borland here, an IEA member, a media evaluation specialist, and the Director of Research Programs at Rockman Et Al. When the nationwide call came in March to help flatten the curve, I had just wrapped up a round of face-to-face playtesting. It had been nerve-wracking to wipe down every surface and hope there weren’t any lingering germs from one family to the next, so I fully embraced the idea of taking my data collection efforts online.
Hot Tips:
- Communicate expectations to participants ahead of time. Let participants know as much as possible in advance. What technology devices are needed? What software is required? Will participants be screen sharing, or will they need a device with a video camera to enable observation? Remind participants to make sure all technology is charged or plugged in.
- Take time to help participants feel at ease. One of the upsides of remote data collection is that it allows participants to be in the comfort of their own homes—but it can still feel strange to have people observing. If you plan to have additional observers, confirm that it is okay ahead of time rather than putting participants on the spot. Ask observers to keep their microphones and cameras off and provide a shared online document where observers can pose questions or note what they see. Spend a few minutes breaking the ice.
- Expect the unexpected. If the upside of participants being able to participate from home is having them be more at ease than they would be in a lab or office setting, the downside is the subsequent lack of control over the testing environment. I don’t have specific tips here, aside from being prepared for a little chaos, and having the willingness to be flexible when something unexpected or disruptive inevitably occurs.
- Pay attention to human subject safety. Safety needs to be the number one priority when conducting online research—especially with youth. I require an adult to be present with children at all times during remote data collection sessions, and I take the utmost care to ensure that I have a secure video connection every time I run an online data collection session. Nonetheless, at the beginning of each session, I explain what I will do if the secure connection is compromised. Thankfully I’ve not had uninvited guests show up, but I feel better having a plan. Your protocols may also need to be updated since online research is essentially in-home research. Decide ahead of time what you will do if you see or hear something that shouldn’t be stored on video or something serious enough that it needs to be reported to another authority. Check with your IRB point-person if you have questions or need further recommendations.
I’ve learned to embrace the joy of online data collection, but I also look forward to a time where I will be able to get back to doing things the pre-COVID ways as well.
The American Evaluation Association is celebrating Indiana Evaluation Association (IEA) Affiliate Week. The contributions all this week to aea365 come from IEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.
Hi Sheila,
I am an educator in Alberta and am currently working towards completing my Professional Masters of Education at Queen’s University with a specialization in Assessment and Evaluation. As a part of our work, we are to connect with the Evaluation Community through this blog. I came across your post while searching for articles around data, as I feel like this is an ongoing area of growth for myself.
Your article resonated with me as I (and many other educators) have continued to navigate the fluctuating terrain of both in-person and online learning; while also ensuring learning is uninterrupted and assessment and evaluation of student learning occurs.
The “Hot Tips” you have posted are both helpful and relevant… even after a year into this pandemic and making the shift online. I appreciate how applicable they are to all evaluation settings. As an elementary teacher, all tips were even applicable to our youngest learners.
I found myself nodding my head as I read your post. Specifically, “pay attention to human subject safety.” This is so important as people are isolated (and continue to have to isolate for periods of time now) and may not realize the implications of their actions.
Furthermore, I appreciated how you highlighted how participants are in the comfort of their own homes. In speaking to my students, they appreciate being home, however highlighted a similar point to what you had said about not enjoying being watched through a camera on their device.
Most importantly, I appreciate how you have highlighted the importance of communication. So often issues arise within an evaluation (and in daily life) due to a lack of or poor communication. Strong communication on the other hand helps develop trust, respect and aids in relationship building. With regards to facilitating an evaluation, when there is strong and clear communication, the data that is collected is more accurate, valid and credible.
Thank you for highlighting such a fantastic concise list of “Hot Tips” and sharing your wisdom.
Michelle
Please note: My above post is directed to Jennifer Borland the author of this article rather than Sheila Robinson who posted the article.
Regards,
Michelle
Hello Jennifer,
My name is Laura Thomson, I am currently a masters student with Queens University in Ontario where I am studying assessment and evaluation in Education. I think the ‘Hot Tips’ you’ve provided is a great way to initiate conversation and the feeling of comfort with the subjects you are collecting data from. This reminds me of my own current situation as I am also a teacher in Ontario experiencing the High School Learn@Home program. As I begin the next Quadmester, these ‘Hot Tips’ are extremely relevant and helpful as I will be able to use these steps to being my course. Giving students or stakeholders expectations at the beginning of a course or data collection, allows them to understand what to expect during your time/my time with them. I particularly admire the step of taking the time to help participants feel at ease. As we are all discovering the online world, I believe it is important to skill establish a connection with the stakeholder/student to ensure that they feel comfortable with what they are doing. Your point about expecting the unexpected is very relevant to the teaching world as it is out of our control for what students are doing while being assessed. Students have access to online information that will help them answer assessment questions which skews natural assessment data. Lastly, your tip about paying attention to the safety of your subject really stood out to me. Students who are online may have specific reasons as to why they are learning at home. Everyone may be fighting a battle we, as teachers, know nothing about, therefore the safety of our students is really important to us. Even though you spoke to a younger audience about having a parent present for supervision, I believe it is important to consider at home life, in case, we as teachers, do witness something online that needs to be reported to authority. I found your post to be very helpful and relatable which has taught me to bring these ‘Hot Tips’ with me as I begin my journey as a Learn@Home teacher with new students this quadmester.