Identifying scammers in online studies by Elizabeth Cook, Makedah Johnson, and Jennifer Manlove

Greetings! We’re Elizabeth Cook, Makedah Johnson, and Jennifer Manlove, researchers at Child Trends, a non-profit research organization in Bethesda, MD. We are evaluating an innovative web-based mobile app designed to prevent teen pregnancy. Since this intervention takes place exclusively online and by text message, we rarely have direct contact with our study participants, so it is essential to maintain integrity in our enrollment processes. As is the case with other tech-based evaluations that provide study incentives, we anticipated scammers, so we developed an approach to identify and remove suspicious participants.

We know from similar studies that identifying suspicious accounts can be a difficult, time-consuming process—but also a necessary one. Since we use social media ads to recruit participants, we need to make sure we only enroll people who meet our eligibility criteria, not just those who claim they do.

We have encountered two types of suspicious screener attempts:

• Scammers: ineligible on first screener attempt but eventually meet criteria after repeated tries
• Duplicates: eligible on first screener attempt and tries to enroll again

Below is a hypothetical example of a scammer who did not meet our age eligibility requirements for the study (ages 18-20) until their third screener attempt (ID 13). This user can be identified by the repeated IP address, and was too young (ID 10), then too old (ID 12); they were finally enrolled after claiming they were 18.

We encourage the study teams of similar online interventions to consider the following tips when developing study procedures.

Design the screener so that it’s difficult to figure out the eligibility criteria.

Structure screener questions to make it difficult to guess the right answer to meet the eligibility criteria. For example, instead of asking for age, we ask for date of birth; we also include a non-relevant question about the respondent’s current state of residence so that ineligible participants might think their ineligibility was related to location. These design elements made the screener more challenging to scam.

Develop a protocol to identify and remove suspicious activity.

We review enrollment data three times a week to identify users with the same or similar key identifiers (e.g., IP address, name, email). Once identified as scammers or duplicates, we deactivate their app accounts and notify them that they are being removed from the study. Although this process is time-consuming, it is necessary to ensure the removal of suspicious accounts.

Remember that some suspicious activity isn’t malicious.

Some users may have encountered issues with logging in to the app and decided to restart at the screener, thus creating a duplicate account. Other participants may forget that they previously participated in the study and then re-enroll months later. To address these duplicates, we allow the first attempt of any duplicate user to remain in the study and remove their subsequent accounts.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.