In this Article
Enable Crawler Authentication
DubBot allows you to crawl pages that require a login to access. First, you must Enable Crawler Authentication for the site.
In DubBot, navigate to the Site's Settings and select the Advanced tab.
Scroll down to Crawler Authentication and check the Enable crawler authentication box.
Authentication Method
Select the authentication method from the dropdown:
Basic - simply provide a username and password
Form - for more complex an advanced logins - typically used if the basic login is having issues
Microsoft - typically used with MS Sharepoint site
Script - an advanced option used by our support team only
Username and Password
Enter the Username and Password created for Dubbot authentication. DubBot employees will not be able to see the password you enter.
Login and Password information for authenticated sites is stored in 256-bit AES encryption, so it is secure.
Login Page URL
Enter the full starting URL of the content you want to crawl/inventory after completing the login.
Username and Password Field Selectors
In these fields, enter the CSS selectors for each field so DubBot can locate the username and password fields to enter the information you provided on the login page.
Need help finding CSS Selectors? Review our help article on Finding a CSS Selector.
Form Submit Button Selector
In the Form submit button selector field, enter the CSS selector for the button to submit the username and password information for validation.
If you have questions, please reach out to our DubBot Support team vial email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!