Outlook: Newsletter of the Society of Behavorial Medicine

Winter 2024

Outlook main page »

Identifying and Minimizing Fraudulent Participants: Strategies and Lessons Learned from an Online Mixed Methods Study

Kristina L. Tatum, PsyD, MS

In 2023, during my postdoctoral fellowship, I was awarded the Bridging the Gap Research Award. This timely pilot award supported formative research related to the influence of social drivers of health on health-related quality of life among adolescent and young adult cancer survivors (AYAs) using online recruitment methods. 

In this work, I used online methods for recruitment (e.g., social media) and data collection, which are cost-effective and allow for flexibility, convenience, and increased accessibility to hard-to-reach populations.1 However, early in the study, I encountered an escalating issue: fraudulent participants (also known as "imposters" or "bots")2,3 seeking financial compensation. I describe my experience with fraudulent activity and outline the strategies I implemented to identify and address fraudulent participants, ensuring data authenticity.

Imposter Participants in An Online Survey Pilot Study

In February 2024, we launched a national online survey (target N=200, via REDcap)4,5 that included an option to complete a qualitative interview (target n=20) via Zoom or phone. Participants first completed an online pre-screener, and if eligible, they proceeded to the full survey. Respondents received a $25 e-gift for completing the survey and an additional $25 e-gift card for the interview. Recruitment was conducted through direct mailings, targeted paid social media advertisements, flyers, institutional shared resource, local health fairs, and emails to AYAs-focused organizations. Participants could access our recruitment webpage and online pre-screener.

Within two months of launching the survey, we received a large number of responses (~487). Early quality checks identified unusual activity, including duplicate surveys completed in a short time, inconsistent verifiable data (e.g., age at diagnosis, phone number), suspicious referral sources (e.g., friend), patterns of duplicate email addresses from the same email provider, short survey completion time (e.g., minutes), and repeated information across multiple entries.6–8 We flagged 100% as likely fraudulent. After consulting with mentors and reviewing the literature,7,8 we developed a protocol to detect patterns suggesting fraudulent participants to minimize their activity and increase quality of data capture.

Overview of Strategies Used to Detect and Address Fraudulent Participants

We modified our approach based on previous studies6-12 and strategies to mitigate these attacks, ensuring data integrity, and maintaining credibility of our findings. We removed eligibility criteria from the study advertisement, thus creating a barrier by not fostering misrepresentation to meet the criteria. We made follow-up phone calls to validate contact information and confirm survey completion. Additionally, we cross-checked pre-screener data with survey responses, including age, cancer type, and email address. We also included duplicate open-ended questions throughout the survey for verification. Regular data checks were conducted to review survey responses for inconsistencies, rapid completion times, and nonsensical responses. Furthermore, we included a clause in our consent form, stating: “Participants will be removed from the study without remuneration in cases of fraud and will not receive additional remuneration for completing the study more than once.”11 After implementing revised strategies to mitigate fraud during the recruitment phase of our study, 142 individuals completed the online survey within 5 months, with 33% identified as likely fraudulent.

The Bridging the Gap Research Award has been instrumental in providing invaluable insights into both my strengths and areas for growth as an early-stage investigator. Addressing recruitment challenges has improved our study outcomes and taught me valuable lessons about the importance of implementing rigorous validity checks when using online recruitment methods.

References:

  1. Evans JR, Mathur A. The value of online surveys. Internet Research. 2005;15(2):195-219. doi:10.1108/10662240510590360
  2. Chandler JJ, Paolacci G. Lie for a Dime. Social Psychological and Personality Science. 2017;8(5):500-508. doi:10.1177/1948550617698203
  3. Dupuis M, Meier E, Cuneo F. Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices. Behavior Research Methods. 2018;51(5):2228-2237. doi:10.3758/s13428-018-1103-y
  4. PA Harris, R Taylor, R Thielke, J Payne, N Gonzalez, JG. Conde, Research electronic data capture (REDCap) – A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics. 2009;42(2):377-81. doi: 10.1016/j.jbi.2008.08.010
  5. PA Harris, R Taylor, BL Minor, V Elliott, M Fernandez, L O’Neal, L McLeod, G Delacqua, F Delacqua, J Kirby, SN Duda, REDCap Consortium, The REDCap consortium: Building an international community of software partners. Journal of Biomedical Informatics. 2019;95:103208. doi: 10.1016/j.jbi.2019.103208]
  6. Levi R, Ridberg R, Akers M, Seligman H. Survey Fraud and the Integrity of Web-Based Survey Research. American Journal of Health Promotion. 2022 Jan;36(1):18-20. doi: 10.1177/08901171211037531
  7. Ridge D, Bullock L, Causer H, Fisher T, Hider S, Kingstone T, Gray L, Riley R, Smyth N, Silverwood V, Spiers J, Southam J. 'Imposter participants' in online qualitative research, a new and increasing threat to data integrity? Health Expectations. 2023 Jun;26(3):941-944. doi: 10.1111/hex.13724
  8. Davies MR, Monssen D, Sharpe H, Allen KL, Simms B, Goldsmith KA, Byford S, Lawrence V, Schmidt U. Management of fraudulent participants in online research: Practical recommendations from a randomized controlled feasibility trial. International Journal of Eating Disorders. 2024 Jun;57(6):1311-1321. doi:10.1002/eat.24085
  9. Roehl, J. M., & Harland, D. J. (2022). Imposter Participants: Overcoming Methodological Challenges Related to Balancing Participant Privacy with Data Quality When Using Online Recruitment and Data Collection. The Qualitative Report. 2022;27(11): 2469-2485. doi.org/10.46743/2160-3715/2022.5475
  10. von Ahn L, Maurer B, McMillen C, Abraham D, Blum M. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science. 2008;321(5895):1465-1468. doi: 10.1126/science.11603796
  11. Pozzar R, Hammer MJ, Underhill-Blazey M, et al. Threats of Bots and Other Bad Actors to Data Quality Following Research Participant Recruitment Through Social Media: Cross-Sectional Questionnaire. Journal of Medical Internet Research. 2020;22(10):e23021. doi: 10.2196/23021
  12. Storozuk A, Ashley M, Delage V, Maloney EA. Got Bots? Practical Recommendations to Protect Online Survey Data from Bot Attacks. The Quantitative Methods for Psychology. 2020;16(5):472-481. doi:10.20982/tqmp.16.5.p472