In this Article
DubBot support must enable this option for the account. If you do not see this option in your Advanced tab, please contact DubBot's Support team via the app's chat or email help@dubbot.com.
Enable Crawler Authentication
DubBot allows you to crawl pages that require a login to access. First, you must Enable Crawler Authentication for the site.
In DubBot, navigate to the Site's Settings and select the Advanced tab.
Scroll down to Crawler Authentication and check the Enable crawler authentication box.
Authentication Method
Select the authentication method from the dropdown:
Basic - for simple logins that present a username and password field in a pop-up
Form - for logins that auto re-direct to another page after the Start URL, on which the username and password are entered
Microsoft - typically used with MS Sharepoint sites
Script - an advanced option for multi-page authentication processes
Username and Password
Enter the Username and Password created for DubBot authentication. DubBot employees will not be able to see the password you enter. We recommend you create a new login specifically for this purpose.
Login and Password information for authenticated sites is stored in 256-bit AES encryption, so it is secure.
Basic Authentication Method
Use Basic Authentication if the site displays a pop-up prompting for a username and password.
Form Authentication Method
Use the Form Method when the Site's Start URL automatically redirects to a page where the user (or crawler in our case) enters a username AND password then selects a button to complete the process and be returned to the Start URL.
Login Page URL
Enter the full starting URL of the page where the username and password fields to login are found.
Username and Password Field Selectors
In these fields, enter the CSS selectors for the username and password fields so DubBot knows where to enter the credentials you provided.
Need help finding CSS Selectors? Review our help article on Finding a CSS Selector.
Form Submit Button Selector
In the Form submit button selector field, enter the CSS selector for the submit button to validate the username and password.
Microsoft Authentication Method
The Microsoft Method is preconfigured to navigate the authentication process common to Microsoft SharePoint sites. You will only need to enter the Username, Password, and Login Page URL for the site.
Script Authentication Method
The Script Method is for situations where a multi-page authentication process is required for a site. For example, on the first page, the username is entered, and a button is selected to continue. On the next page, the password is entered, and a Submit or Validate button is selected, and the user is then authenticated and returned to the site's home page.
To enable our crawler to move through the process, each step needs to be created to ensure success.
The Login script steps dropdown offers the following options:
Click - use this when you need to select a button and enter the unique CSS selector for the button
Fill field - used to enter information into a field. Has dropdown options of Username, Password, and Custom. For each, you must enter the unique CSS selector for the field. Username and Password get the field's value from the Username and Password fields located above the Login steps. Custom fields must have both the CSS selector and the text to fill entered.
Go to URL - full URL, including https://, of the page you want the crawler to visit
Press -
Type -
Wait for milliseconds - instructs the crawler to wait for the entered number of milliseconds. Most often used to allow time for a page to fully load before any action is taken.
Wait for selector -instructs the crawler to wait until a particular CSS selector is loaded on a page before continuing the login. Used to allow time for the page to load before taking action.
Wait for URL - instructs the crawler to wait for a particular URL to load before continuing with the login steps.
In the screenshot below, a sample demonstrates what Scripted Authentication may look like. Keep in mind, this is only a sample; there are many different variations possible, and this needs to be constructed with your particular instance in mind.
Scripted logins can get a bit complicated, so feel free to reach out to us via the chat bubble in the lower-right corner of the screen or send us an email at help@dubbot.com .
The Broken Link Checker does not support authentication, so links requiring the authentication configured on the site will most likely return a Broken Link reason of 403 (Unauthorized) or similar.
Learn more on Crawling a Development Site.
If you have questions, please reach out to our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!