This is an Advanced article. If you are having issues with DubBot crawling your site, feel free to reach out to support via our chat or send an email to email@example.com.
The DubBot user agent is the following:
Mozilla/5.0 (compatible; Dubbotbot/0.2; +http://dubbot.com)
Website administrators often detect and “block” access to websites based on a website visitor's detected user agent. The user agent is the technical specification of the website browser used to access the website. Some websites are configured to detect and block user agents suspected of being crawlers or bots. This layer of security can result in preventing DubBot from crawling a site. Captcha is an example of this detection process.
DubBot Tools to change User Agent
DubBot has added tools in the advanced Setting when configuring a Site for crawling that allow administrators to mask the default DubBot User Agent if the DubBot crawler is being blocked.
Test the user agent
Chrome's User-Agent Switcher for Chrome plug-in can be configured to allow you to test whether your website can be accessed using DubBot's user agent. To use this plug-in, you will need to add the details of DubBot's User Agent to the plug-in. You can then visit the website that you are configuring for crawling. If the website presents an additional security screen, like a prompt to enter CAPTCHA information, it is likely that the website is blocking DubBot from crawling by detecting the User Agent.
Changing the designated User Agent
On the Advanced tab of the Site's settings, you can change the declared User Agent to your browser's declared User Agent or any of the most commonly used User Agents. An example of a widely used User Agent that you can change to is the following:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/18.104.22.168 Safari/537.36
Other popular user agents: Latest User Agents
If you do not know how to determine your User Agent, this webpage will easily provide those details: What is my User Agent?