Hi, Jennifer Borland here, an IEA member, a media evaluation specialist, and the Director of Research Programs at Rockman Et Al. When the nationwide call came in March to help flatten the curve, I had just wrapped up a round of face-to-face playtesting. It had been nerve-wracking to wipe down every surface and hope there weren’t any lingering germs from one family to the next, so I fully embraced the idea of taking my data collection efforts online.
- Communicate expectations to participants ahead of time. Let participants know as much as possible in advance. What technology devices are needed? What software is required? Will participants be screen sharing, or will they need a device with a video camera to enable observation? Remind participants to make sure all technology is charged or plugged in.
- Take time to help participants feel at ease. One of the upsides of remote data collection is that it allows participants to be in the comfort of their own homes—but it can still feel strange to have people observing. If you plan to have additional observers, confirm that it is okay ahead of time rather than putting participants on the spot. Ask observers to keep their microphones and cameras off and provide a shared online document where observers can pose questions or note what they see. Spend a few minutes breaking the ice.
- Expect the unexpected. If the upside of participants being able to participate from home is having them be more at ease than they would be in a lab or office setting, the downside is the subsequent lack of control over the testing environment. I don’t have specific tips here, aside from being prepared for a little chaos, and having the willingness to be flexible when something unexpected or disruptive inevitably occurs.
- Pay attention to human subject safety. Safety needs to be the number one priority when conducting online research—especially with youth. I require an adult to be present with children at all times during remote data collection sessions, and I take the utmost care to ensure that I have a secure video connection every time I run an online data collection session. Nonetheless, at the beginning of each session, I explain what I will do if the secure connection is compromised. Thankfully I’ve not had uninvited guests show up, but I feel better having a plan. Your protocols may also need to be updated since online research is essentially in-home research. Decide ahead of time what you will do if you see or hear something that shouldn’t be stored on video or something serious enough that it needs to be reported to another authority. Check with your IRB point-person if you have questions or need further recommendations.
I’ve learned to embrace the joy of online data collection, but I also look forward to a time where I will be able to get back to doing things the pre-COVID ways as well.
The American Evaluation Association is celebrating Indiana Evaluation Association (IEA) Affiliate Week. The contributions all this week to aea365 come from IEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to firstname.lastname@example.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.