All Collections
Release Updates
2021
Crawling behind a login
Crawling behind a login

February 6, 2021

Updated this week

DubBot allows you to crawl pages that require a login to access.

If you find that this feature is unavailable in your Advanced tab, please contact DubBot's Support team via the app's chat, or you can email help@dubbot.com.

Using your authentication method, create a user account and login for DubBot to access your secure content.

In DubBot, navigate to the Advanced tab under Settings:

DubBot site settings advanced tab

Scroll down to Crawler Authentication and check the Enable crawler authentication box.

DubBot App Crawler Authentication option displayed.

There you will find 3 options to select for your authentication method: Basic, Form, or Microsoft.

DubBot App Crawler Authentication options for Authentication Type.

Finally, enter the information in the fields displayed and save your changes.

DubBot App Crawler Authentication options displayed for Basic Authentication Method.


If you have questions, please reach out to our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!

Did this answer your question?