DubBot now allows you to crawl pages that require a login to access.
Using your authentication method, create a user account and login for DubBot to access your secure content.
In DubBot, navigate to the Advanced tab under Settings:
Scroll down to Crawler Authentication and check the Enable crawler authentication box.
There you will find 3 options to select for your authentication method: Basic, Form, or Microsoft.
Finally, enter the information in the fields displayed and save your changes.