In this Article
What is a Crawl Warning?
Are you seeing a Crawl Warning alert on your Site or Page Set's dashboard?
This alert appears on the Recent Activity panel of a Site's dashboard if a crawl of a site is not successfully completed.
Find the Crawl Log
On a Site's Dashboard, in the Site Overview Panel, select the link shown with the date of the Latest Crawl to access that Crawl Log.
Review the Crawl Log
Once in the Crawl Log, check both the Status and Response Code columns. Here you will find clues to resolving your Crawl Warning issues.
403 Response Code
The 403 Code occurs when the target server understands the request but refuses to authorize it. This error generally means that DubBot's Crawler IP addresses need to be allow-listed on your web server.
404 Response Code
A 404 code indicates that our crawler cannot find the starting URL or site map set up for the site in question. You should review the Site Settings and load the starting page or site map into a browser yourself. If the URL given in the settings is redirecting to another URL, you should update the Site Settings to the actual URL.
Some common errors that result in 404s:
Adding or removing a
wwwat the start of the starting URL for the Site.Confusing
httpandhttpsin the starting URL. It matters.
Ignored Status: Has a Query String
If you see the status Ignored and see Reason: Has a Query String when mousing over the ?, you must Modify the Site Settings and uncheck the Ignore Pages with URL Queries box.
Enabling queries can sometimes create very large page inventories. If you see this happening for the site, contact us for help at help@dubbot.com or the in-app chat.
Error Status: Navigation Timeout
If you encounter a Navigation Timeout error there are two options:
If the error states "Navigation timeout of 60000ms exceeded," the page is taking longer than 1 minute to load and presents as an error. Visit the Advancedtab of the Site Settings and edit the Page Load Timeout (in seconds) to be 120. If after re-crawling, you still see the Crawl Warning, you will need to contact us for help.
If the error states "Navigation timeout of 1200000ms exceeded," contact us for help at help@dubbot.com or the in-app chat.
Site Map Crawls
If the site is configured as Sitemap crawl, the starting URL must be an xml sitemap. The app doesn't crawl HTML sitemaps. Contact us for help at help@dubbot.com or the in-app chat and we can help.
Ensure the sitemap actually contains pages.
If you have questions, please contact our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!