DubBot Staff

DubBot inventories website content by going to a URL that is supplied to the application and essentially clicking on every link within that webpage. If the link is contained within the same domain specified in the Site settings, DubBot will download a version of the page for analysis. 

DubBot will then click on every link within that page to continue traversing and downloading versions of webpages throughout your website. This process continues until there are no links remaining that have been identified within the provided domain.

Crawling a website is a common practice for Search Engines when performing analysis of webpages for search engine ranking.

A common analogy is that crawling a website is like a spider crawling a web.

Please note: DubBot can also be configured to crawl a specified list of URLs via manual upload, CSV upload, or supplied Sitemap. In these cases, the crawler will only inventory the links provided in those documents or explicitly maintained. Learn more about <a href="https://help.dubbot.com/en/articles/3204505-site-crawler-options">Site Crawler options</a>.

<a href="https://help.dubbot.com/en/articles/6405910-removing-accessibility-checks">Removing Accessibility Checks</a>

<a href="https://help.dubbot.com/en/articles/6007939-resolving-an-issue-versus-ignoring-an-issue">Resolving an Issue versus Ignoring an Issue</a>

<a href="https://help.dubbot.com/en/articles/6101985-using-libguides-by-springshare">Using LibGuides (by SpringShare)</a>

<a href="https://help.dubbot.com/en/articles/5952665-why-is-my-dubbot-site-indexing-urls-outside-of-our-group-or-domain">Why is my DubBot site indexing URLs outside of our Page Set or domain?</a>

<a href="https://help.dubbot.com/en/articles/5905953-what-is-readability-why-does-it-matter">What is Readability? Why does it matter?</a>

<a href="https://help.dubbot.com/en/articles/4708329-issue-not-found-in-current-view">Issue not found in current view</a>

<a href="https://help.dubbot.com/en/articles/4257010-color-coding-for-scores">Color coding for scores</a>

<a href="https://help.dubbot.com/en/articles/4222940-issue-not-highlighting-on-the-webpage-when-selected">Issue not highlighting on the webpage when selected?</a>

- <a href="https://help.dubbot.com/en/articles/6405910-removing-accessibility-checks">Removing Accessibility Checks</a>
- <a href="https://help.dubbot.com/en/articles/6007939-resolving-an-issue-versus-ignoring-an-issue">Resolving an Issue versus Ignoring an Issue</a>
- <a href="https://help.dubbot.com/en/articles/6101985-using-libguides-by-springshare">Using LibGuides (by SpringShare)</a>
- <a href="https://help.dubbot.com/en/articles/5952665-why-is-my-dubbot-site-indexing-urls-outside-of-our-group-or-domain">Why is my DubBot site indexing URLs outside of our Page Set or domain?</a>
- <a href="https://help.dubbot.com/en/articles/5905953-what-is-readability-why-does-it-matter">What is Readability? Why does it matter?</a>
- <a href="https://help.dubbot.com/en/articles/4708329-issue-not-found-in-current-view">Issue not found in current view</a>
- <a href="https://help.dubbot.com/en/articles/4257010-color-coding-for-scores">Color coding for scores</a>
- <a href="https://help.dubbot.com/en/articles/4222940-issue-not-highlighting-on-the-webpage-when-selected">Issue not highlighting on the webpage when selected?</a>

DubBot crawls your website to inventory your content. What do we mean by crawling?

More Frequently Asked Questions