The Ignored Paths for Specific Sites article has the most up-to-date information on this topic.
DubBot now allows administrators to ignore webpages utilizing a regular expression. Webpages or folders will be excluded from site crawl based on the regular expression added.
If you need help utilizing this new feature, reach out to DubBot support using the in-app chat or email help@dubbot.com.
Learn more about Ignoring Paths for Specific Sites.