DubBot now allows you to crawl pages that require a login to access.

Using your authentication method, create a user account and login for DubBot to access your secure content.

In DubBot, navigate to the Advanced tab under Settings:

DubBot site settings advanced tab

Scroll down to Crawler Authentication and check the Enable crawler authentication box.

DubBot App Crawler Authentication option displayed.

There you will find 3 options to select for your authentication method: Basic, Form, or Microsoft.

DubBot App Crawler Authentication options for Authentication Type.

Finally, enter the information in the fields displayed and save your changes.

DubBot App Crawler Authentication options displayed for Basic Authentication Method.

Did this answer your question?