Sites: Crawling and Troubleshooting
Fine tuning and working through potential crawl issues.
12 articles
- Crawling a Development SiteHow to utilize DubBot's authentication settings to crawl a dev site.
- Using LibGuides (by SpringShare)
- Why aren't my PDFs being found?Troubleshooting PDFs missing from a site's inventory.
- Troubleshooting Crawl WarningsTips to get your sites crawling again.
- Filtering crawler from Google Analytics 4 (GA4) Reporting & DubBot's IP AddressFilter DubBot's crawler from Google Analytics 4 reports
- Example Custom User Agent
- What is robots.txt and why do I need to update my site settings?
- Your 404 page needs a 404 page statusWhy this is critical for both a successful crawl and accessibility
- Crawling Behind a Login
- Sidearm Sports SitesTips for managing these athletic third-party sites.
- Use the Power of Caching to Speed up Site CrawlsUsing ETags for faster crawls.
- How to Enable Etags