With DubBot, you can configure automated accessibility testing for your web development site, in most cases, even if it's behind a login.
In this article ...
Robots.txt
A robots.txt file is a tool used by websites to tell search engines which parts of their site should not be explored or indexed.
For DubBot to access your development site, you might need to ignore robots.txt. To do this, navigate to Settings and then to the Advanced tab in the DubBot app. Uncheck Obey robots.txt.
For more information, check out our Help Article all about robots.txt.
DubBot's Static IPs
If your development site remains inaccessible to the DubBot crawler, it may indicate that our crawler is being blocked. To address this, your server administrator can allow-list our crawler with the following static IPs and user agent.
The static IP addresses for our crawler are 34.213.65.175 and 52.36.131.191.
If your account uses the Uptime Monitor, you should also include 52.39.13.97 .
Our user agent is Dubbotbot .
Crawling Authentication
If your development site requires a username and password to login, DubBot may still be able to crawl by enabling crawler authentication. If this setting is not available on your Advanced tab in Site settings, contact DubBot Support to have it set up in your organization's account.
You can find more details on how to use crawler authentication with DubBot in our Crawling Behind a Login help article.
The Broken Link Checker does not support authentication, so links requiring the authentication configured on the site will most likely return a Broken Link reason of 403 (Unauthorized) or similar.
Still Need Help?
If you have questions, please reach out to our DubBot Support team vial email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!