DubBot now allows administrators to ignore webpages utilizing a regular expression. Webpages or folders will be excluded from site crawl based on the regular expression added.
If you need help utilizing this new feature, reach out to DubBot support using the in-app chat or email help@dubbot.com.