Allow-list is a term used in the field of computing and network security to describe the practice of specifying authorized entities, such as software applications or network addresses, which are granted access privileges to a website or server. Typically this is configured with tools like firewall rules or other access control models.
Formerly, the term "white-list" was used to describe this process. In an effort to eliminate oppressive language it was revised to "allow-list".
It is not always necessary to explicitly allow-list DubBot in order to crawl a website. If the crawler is having trouble inventorying pages, is timing out frequently on otherwise healthy pages, or has other issues with a crawl it may be a sign that DubBot's static IP should be added to your organizations allow-list. This is usually done by an IT team or whoever manages your website's or network's security.
There are other ways to manage DubBot's access to your website. Learn more about that in this article about Robots.txt.
Still Need Help?
If you have questions, please reach out to our DubBot Support team via email at help@dubbot.com or via the blue chat bubble in the lower right corner of your screen. We are here to help!