Hello! We are Laura Beals and Barbara Perry, internal evaluators at Jewish Family & Children’s Service, a large social-service agency located outside of Boston, MA. We have been gathering feedback from clients and other constituents via surveys, often online, for many years, but we recently completed a needs assessment project in which we also wanted to gather information from potential clients, their families, and professionals. One way we attempted to reach these new populations was through social media (via the agency’s Facebook page) and a Massachusetts non-profit email listserv.
In our invitation to participate in the survey—the Facebook post or the email invitation to the listserv—we clearly but briefly outlined who we were looking for (the target population), what we were asking them to do, what the incentive was, and how to access the link. In the introduction to the survey itself, we more thoroughly outlined participation expectations and indicated that we would (snail) mail the first 100 participants who were interested a $5 gift card to Dunkin Donuts. We had decided to use a physical gift card, instead of an electronic gift card, and we are so glad that we did…because about two weeks after our first survey completions from the Facebook and listserv posts, we started to get back envelopes of gift cards with bright yellow “return to sender” stickers.
This prompted us to immediately take a closer look at these responses—we found email addresses that didn’t match names provided in the mailing address (not that they have to, but it was suspicious), many email addresses had random numbers at the end, all had out-of-state mailing addresses, searching on whitepages.com suggested these likely were fake people, they were all in the same survey collector, and the content of their responses were not in alignment with those that we knew from real people (e.g., they completed the closed-ended questions, but none of the open-ended). Luckily, we were able to determine a pattern to these fraudulent responses and eliminate them from our analysis. We had planned to look at our responses for duplicates (email address, IP addresses if known, etc.), but had not thought about reviewing for fraud!
As more-and-more evaluators may be turning to social media to gather survey responses, especially for youth, we wanted to share three lessons learned from our experiences:
- Use a unique collector for each distribution type. We use SurveyMonkey as our surveying tool, and it gives us the ability to create distinct links for each type of distribution (email to clients, post to Facebook, email to listserv, etc.). This allowed us to pinpoint which collector was the culprit.
- Read “Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and Tradeoffs.” The authors provide a table of “Methods of Detecting and Preventing Internet Study Duplication and Fraud and Their Implications.”
- Include the step of examining your responses for signs of fraud before beginning analysis.
Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to email@example.com. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.