According to Goodrich et al., (2023) the quality of social science data is fundamental. High quality data depends on a well-built data collection tool and adequate participant recruitment. The practice of study recruitment via social media may allow access to a wider audience; unfortunately it also has a downside – BOTS. The use of incentives is another common practice with online surveys, but it can increase the likelihood that a study is inundated with fraudulent responses originating from BOTS. They have become an increasing problem as they are able to get through CAPTCHA, bot detectors, duration to take survey, and IP addresses. It can usurp a vast amount of time to comb through your data to clean it up and remove the bot-generated responses.
Goodrich et al., (2023) recommend a multifaceted approach to bot prevention. Among their recommendations to combat bots is with the use of a Honeypot Question. This is a question that is hidden from human participants, but seen by bots. When incorporated into your survey, you can see which “participants” answered your honeypot question to recognize them as bot responses. It does not prevent bot responses, but it is easy enough to incorporate into your survey to ease your identification of bot-generated survey fraud. Below I will walk you through the steps to set up a Honeypot Question in Qualtrics.
Note, the javascript below only works when “New Survey Taking Experience” is toggled off in Survey Options –> General.
First create your survey, then add an additional block for your Honeypot Question. This question doesn’t necessarily have to be as obvious as the example below, but could be something like ‘Where in the United States are you?’ or ‘What do you like to do for fun?’ or ‘Do you like cheese?’
Next you will add JavaScript to make this question only visible to Bots.
Scroll to the bottom of this section and click on JavaScript.
Below is what you will add into the JavaScript box:
- You will enter the following text after the word (function() { THE TEXT BELOW WILL GO HERE BETWEEN THESE BRACES/PARENTHESES }):
jQuery(“#”+this.questionId).prev(‘.Separator’).hide();
jQuery(“#”+this.questionId).hide();
and then you will add the following to the second section.
- (function() { THE TEXT BELOW WILL GO HERE BETWEEN THESE BRACES/PARENTHESES }):
jQuery(“#NextButton”).click();
It will look like the picture below. You can leave the remaining JavaScript as is.
Next, we will modify the survey flow to ensure that the when the Bot question comes up, it will direct them to the end of the survey.
As in the picture below you will click on survey flow.
When you open Survey flow it will look something like this depending on how many blocks you have.
As you can see you should have a block just for the BOT question.
You will first click on “Add Below” on the bot question block. As you can see in the picture below several options will pop up. You will select the “Branch” option.
Once you select the branch option you will need to create a condition. You will select –
- Select the question from the first drop down menu.
- Next select the Bot question from the second drop down menu.
- Then you will select the answer for this question from the third drop down menu.
- Lastly you will select ‘is displayed’ from the last drop-down menu.
- Hit ok to save.
Lastly you will click “add new element here” after the branch and select “End of Survey”.
Once you have completed all of these steps you will hit “Apply” in the lower right-hand corner to save changes.
You can test your survey, and you will know it is working if the “Bot” question does not come up for you. It will be hidden and only appear for Bots due to the JavaScript coding we inserted.
We hope that this quick addition can help weed out the Bot answers from your data.
Happy Surveying!
References
Goodrich, B., Fenton, M., Penn, J., Bovay, J., & Mountain, T. (2023). Battling bots: Experiences and strategies to mitigate fraudulent responses in online surveys. Applied Economic Perspectives and Policy, 45(2), 762-784. https://doi.org/10.1002/aepp.13353