More information for you in two places: Crawl Warnings and Redirected URLs.
Expanded Crawl Warning Information
Site and Page Set History panels now show more information about why a particular site isn't crawling. The Crawl Warning alert now includes the reason the crawl failed.
Some of the reasons that a site may not successfully crawl include:
Reason: Disallowed by robots.txt
Reason: Not in domain
Reason: Response not OK
Reason: Redirected to XYZ URL
Reason: Not in hierarchy
Reason: Unable to load page: Timeout 60000ms exceeded.
Learn more about Troubleshooting Crawl Warnings.
Redirected URLs now Available for Sites
The References Panel on a Site Dashboard now has a filter option to show Redirected URLs for the site.
References filtered to Redirected URLs provides the following information:
URL and a link to open the URL in a new window
Reference Type
Response Code
Content Type
Filesize
You can also download this content as a CSV file using the Download CSV button, shown by the cloud icon.
Learn more about the Site or Page Set Dashboard.
If you have questions, please reach out to our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!