In this Article
Pro Tip: The vast majority of Crawl Warnings can be resolved by following these steps:
Open the Site's Settings and copy the Start URL.
Paste that URL into your browser of choice.
Compare the URL of the page that loads in the browser with the URL in the settings.
Copy and paste the loaded URL into the site's settings.
You can then either wait for the next scheduled crawl, recrawl the site yourself, or contact us for help with the recrawl or any other questions.
What is a Crawl Warning?
Crawl Warnings are generated when the DubBot crawler does not successfully crawl a site. Several reasons can cause this. To help with your troubleshooting, these Crawl Warning alerts also include the reason the crawl failed. Several of the reasons are addressed below with guidance on how to fix the error.
The Crawl Warning alerts can appear on a Dashboard and in the Recent Site/Page Set Activity Panel.
Disallowed by robots.txt
If a crawl fails due to being Disallowed by robots.txt, there are two options:
You can contact the person who can edit the robots.txt file for the site and ask for them to allow list the following: The static IP addresses for our crawler are 34.213.65.175 and 52.36.131.191. If your account uses the Uptime Monitor, you should also include 52.39.13.97 . They should also allow our user agent, Dubbotbot .
By default, sites will obey robots.txt rules configured for a site. To ignore the settings in the robots.txt file, the Obey Robots.txt box should be deselected in a site's Advanced Settings.
Not in domain
The Not in domain reason most often occurs when a site's Start URL redirects to a different URL, on a different domain. The domain part of the URL is the part between https:// and the next forward slash. For example, in the URL https://www.college.edu/intranet, the domain is www.college.edu.
For example, if a Start URL: https://www.college.edu/intranet redirects to https://intranet.college.edu, the crawl will not continue because the redirected URL does not match the original Start URL.
Simply update the site's Start URL and you should be ready to go.
Response not OK
A crawl can return a Response not OK for several reasons. For more information about the issue behind this response, review the Crawl Log for the site's last crawl. Review Troubleshooting Crawl Warnings for more help.
Redirected to XYZ URL
The Start URL for the site redirects to a URL that is not within the site's original Start URL. DubBot will not follow redirects unless the new location falls within the original Start URL's structure.
For example: A Start URL of https://www.subdomain.college.edu redirects to https://admissions.college.edu. The redirected URL will not be followed because the new URL does not fall under the original URL that started with www.subdomain.college.edu.
Not in hierarchy
The easiest way to check a Not in hierarchy reason is to copy the configured Start URL for a site into a browser and compare the loaded page URL with the configured URL.
Unable to load page
If the crawler can't load the Start URL within the site's configured time, it returns the Unable to load page reason. Find the site's Start URL and try to load that page in a browser. If the page won't load, contact the people responsible for the site's back-end maintenance and keeping it online and ask for a status update.
More information on
If you have questions, please reach out to our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!