Crawling behind log in

February 6, 2021

Updated over a week ago

DubBot now allows you to crawl pages that require a login to access.

Using your authentication method, create a user account and login for DubBot to access your secure content.

In DubBot, navigate to the Advanced tab under Settings:

DubBot site settings advanced tab

Scroll down to Crawler Authentication and check the Enable crawler authentication box.

If this is unavailable in your Advanced tab, please contact DubBot's Support team via the app's chat, or you can email

DubBot App Crawler Authentication option displayed.

There you will find 3 options to select for your authentication method: Basic, Form, or Microsoft.

DubBot App Crawler Authentication options for Authentication Type.

Finally, enter the information in the fields displayed and save your changes.

DubBot App Crawler Authentication options displayed for Basic Authentication Method.

Did this answer your question?