All Collections
Setting Up DubBot (for Administrators)
Additional Administrative Information
Your 404 page needs a 404 page status
Your 404 page needs a 404 page status

Why this is critical for both a successful crawl and accessibility

Updated over a week ago

What is page status?

Page status is a shorthand way to refer to the HTTP status code a web server returns when a browser attempt to load a web page.

What do all those numbers mean?

There are quite a lot of possible codes that a web server can return when trying to load a web page. The codes of importance to us here are:

  • 404: the server cannot find the URL that was requested, "File Not Found".

  • 200: OK status, the page was found at the specified location

Want to see all the codes? All of the possible HTTP status codes

Why is it important?

When a user requests a nonexistent webpage, your server should return an error page with a page status of 404 to let the user know the page is no longer available.

The 404 page status does quite a few things for you:

  • DubBot specific: Lets our crawler know that the page does not exist so it will not be crawled, or inventoried for your site.

  • Informs the user when pages do not exist

  • If you return a 404 status, the screen reader will relay this information upon page load to the website visitor.

  • Returning that 404 status code lets the search engines know that this page no longer exists and needs to be removed from your index. Search engines will no longer link to non-existent pages.

If your server's 404 page returns a 200 (OK) status, here is what happens:

  • DubBot specific: Our crawler will not know that the requested page did not exist and will continue to crawl past your 404 page.

  • A 200 OK status implies that the page was found and the request was successful, which is not the case.

  • The user may unnecessarily spend time traversing the page to get to the content to then find out that the page is simply a 404 page.

  • Returning a 200 and not a 404 is an accessibility issue and an issue for search engines.

  • It can cause unnecessary page traversing for your users, particularly users that utilize a screen reader while browsing.

  • Lawsuits have cited this problem as a significant factor contributing to a website’s overall accessibility.

How do I test it?

You don’t even need DubBot to test your 404 pages' status. We’ve found this third-party (free) service useful for quick testing. We’d recommend testing a URL you know does not exist on your website and verify that a proper 404 status is returned. You can see in the below screenshot where we used 'dubbot.com/url-that-doesnt-exist ' as our test. The second red arrow points to a 404 result.

Screenshot of a test for a Page Not Found error. Top of the screenshot is the URL <a href="https://dubbot.com/url-that-doesnt-exist" target="_blank" rel="nofollow noopener noreferrer">https://dubbot.com/url-that-doesnt-exist</a>. Below that is red arrow pointing to a large blue button with white text that reads Check Status. Below that is a second red arrow pointing to a small orange rectangle with white text that indicates the test result which is "404."

A Quick Win for DubBot Users

While it is important to provide a friendly resource when a link breaks, it should be a top priority for website administrators to find and fix broken links as quickly as possible. In DubBot, you can filter the list of pages that contain broken links to only show pages with a link leading to a 404 error. This prioritizes updates to pages with links that lead to a 404 response, a simple project any website editor can undertake with DubBot.

Did this answer your question?