What are Crawl Errors?

Crawl errors are issues reported by search engines spiders. These issues are reported as they crawl your website and encounter specific problems. Everything the search engine knows about your website starts with your sitemap. Your sitemap contains all the pages & posts and related information (tags, meta, categories, etc). This is how “spiders” learn about your website, scanning or “crawling” your publicly available resources. This is a fundamental process for search engines, so interacting with your sitemap is extremely important. There are different types of errors and some are frivolous, while others are very severe issues.

4xx Client errors | intended for situations in which the error seems to have been caused by the client. Commonly encountered examples of these error types are:

401 Unauthorized – A generic error message, given when no more specific message is suitable.

403 Forbidden – The request was valid, but the server is refusing action.

404 Not Found – The requested resource could not be found but may be available in the future.

414 URI Too Long – The URI provided was too long for the server to process.

5xx Server errors | indicate a server misconfiguration and should be resolved immediately. Commonly encountered examples of these error types are:

500 Internal Server Error – When authentication is required and has failed or has not yet been provided.

501 Not Implemented – The request method is not supported by the server and cannot be handled.

502 Bad Gateway – An HTTP status code that means that one server received an invalid response.

503 Service Unavailable – An HTTP status code that means the server is simply not available right now.

504 Gateway timeout – The server did not receive a timely response to deal with your HTTP request.

A full list of Hypertext Transfer Protocol (HTTP) response status codes can be found on wikipedia here.