Skip to main content

Crawl Logs

Review what occurred during a site's crawls.

​

In this Article

Where to find the Crawl Log

Information on the Latest crawl for the site is found through the link at the bottom of the Site Overview panel on a site dashboard.

Following the link shown by a date beside Latest crawl leads to the Crawl Log.

The Site Overview panel highlighting the date of the latest crawl that links to the crawl log.

Crawl Log Header Information

The Crawl Log header contains the following information:

  • Status - current status of Crawl > Crawling/ Processing / Done / Cancelled

  • Total URLs Identified - total number of pages the crawler found before removing external links or ignored pages configured for the site

  • Pages Indexed- number of traditional web pages that were indexed, or inventoried in the crawl.

  • Assets Indexed - number of supplemental files inventoried in the crawl.

  • Retries - number of times the crawler tried to recrawl any page when the target web server stopped responding

  • Errors - errors encountered by the crawler on this crawl

The numbers shown below these headings as buttons or links will direct you to a listing of those items in the Crawl Log page body below.

There are also arrows to the far left and right of the header to view the previous or next crawl's data.

Sample Crawl Log header showing the crawl's Status, Total URLs Identified, Pages Indexed, Assets Indexed, Retries and Errors.

Breadcrumb Filters

The Crawled URLs breadcrumb allows you to filter the information shown from the latest crawl:


​All URLs for a filter based on the URL state: (selectable in the Select a filter for URLs list)

  • Queued - pages currently in the Queue, waiting to be analyzed

  • Crawling - pages currently crawling

  • Processing - pages currently running through all DubBot checks

  • Crawled- pages that have been successfully crawled. When selecting Crawled, a secondary filter appears: All Types

    • Page - show only traditional web pages

    • Asset - show only PDF files

  • Error - pages that resulted in an error with the crawler, and error codes returned

  • Skipped - pages skipped by the crawler, that the crawler is not going to index, like 404 pages and pages not in the scope of the site

  • Ignored - URLs ignored by the crawler due to issues such as ignored paths or disabled query strings

In the Crawl log, a red box is highlighting the Crawled URLs with dropdowns labeled Crawled and All Types.

Information per URL

Each Filtered URL list shows the following information:

  • URL

  • Status (hovering over these items will give more detail, if available)

  • Response Code

  • Created and Updated date (time shows when hovering over the date)

Example of Crawl Log Panel

View Historical Crawl Logs

Locate the View all crawls button in the top-right of the Latest Crawl log.

The Site's Crawl History Panel displays the following information:

  1. Date and time the crawl Started

  2. Date and time the crawl and processing Finished

  3. Duration of the crawl

  4. Status of the crawl activity

  5. Retries - number of times the crawler tried to recrawl any page when the target web server stopped responding

  6. Total URLs Identified - total number of pages the crawler found before removing external links or ignored pages configured for the site

  7. Number of Errors the crawler encountered when trying to load pages

  8. Pages Indexed - number of pages inventoried/indexed by the crawler

  9. Assets Indexed - number of PDFs indexed by the crawler

  10. Updated - date and time last updated

Select the View all crawls button at the top-right of the Latest Crawl Panel

View all crawls Panel

Learn more about the Site or Page Set Dashboard.

If you have questions, please reach out to our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!

Did this answer your question?