What is a 404 error and why does it occur?
The 404 or Not Found error occurs when the page the user is trying to navigate to cannot be found. Let's take a closer look at the process of this error:
Click on a link: The user clicks on a link on a website or enters a URL in the browser's address bar.
Sending a request to the server: The browser establishes a connection to the server, which is the physical computer on which the files and databases of the site are stored.
Server response: The server sends the browser an encoded response with a three-digit code that shows how the server processed the request.
If the connection to the server is not established, the message "unable to access the site" appears.
If the connection is established and the requested page exists, the browser receives a response with a 200 OK code, which means that the request has been successfully processed and the page has loaded. If the page is not found, the browser displays a 404 error screen.
The reasons for the 404 Not Found error may be as follows:
The page has been deleted.
Broken link: The URL that the user clicked on previously existed, but was changed or entered with a typo, which is why the browser cannot find the page on the server.
Problems on the server: In rare cases, the server may experience failures that cause similar errors.
What is a false 404 error and why does it occur?
False appearance error 404 occurs when the browser receives a successful 200 OK response, but the 404 Not Found message is still displayed or the user sees a page with insufficient content or an empty screen.
The reasons for a false 404 error may be as follows:
Incorrect configuration of the response code: The site owner can replace the 404 error message with a stub page with the 200 OK code for non-existent pages.
Insufficient amount of content: There may be insufficient materials on the page or they may be completely missing.
No access to resources: File robots.txt The site's root directory contains information for search engines about which resources can be scanned. If access to the page or its files is blocked in robots.txt this may cause a Soft 404 error.
"Heavy" page: If there are too many files on the page or their size is too large, the page may take a long time to load, which eventually leads to an error.
Why are 404 errors harmful to SEO?
The drop in the site's rating is not so much due to the presence of 404 pages, but rather to external URLs that point to these pages. A large number of such addresses indicates the low quality of the site, which negatively affects its indexing and positions in search results. In addition, this situation causes negative reactions from users, which manifests itself in an increase in bounce rates and a decrease in the time of interaction with the resource.
It is also worth noting that 404 errors lead to irrational use of the crowling budget that every search engine has. Crawlers scanning website pages have a limited number of URLs that they can visit at a time. However, the presence of 404 errors leads to this budget being wasted, as bots bypass incorrect URLs instead of important pages.
How to find internal and external links to 404 pages
Broken links can be both internal and external. Internal links are located on the site itself and lead to other internal pages of the same site, while external links are located on third-party resources and direct to the pages of your site.
To detect both types of links, you can use free services such as Yandex.Webmaster, Google Search Console, Xenu's Link Sleuth, shareware Screaming Frog SEO Spider Tool, as well as a paid tool will help Netpeak Spider does this in a few clicks.
To detect internal links in Yandex.Webmaster, you need to add the URL of your site, then go to the Links section and select Internal Links. To find such external links, you also need to add the URL of the site, then go to "Links" → "External links" → "Current links" and select the "Broken links only" option.
In Google Search Console, to check the site for 404 errors, you need to add the URL of the resource, then go to the "Coverage" section and filter the pages by the "Excluded" status. As a result, pages with error types "Error 404" and "Not found (404)" will be displayed.
Xenu's Link Sleuth is a program for searching for internal broken links that is easy to set up and use. It should be noted that this application has not been updated for a long time, and it may not work correctly on some devices.
To search for broken pages in Xenu's Link Sleuth, follow these steps: click on "File" → "Check URL" → enter the URL of the desired site → check the box "Check external links" → click "OK". After that, a report will appear in which 404 errors will be marked with the status "not found".
Screaming Frog SEO Spider Tool is one of the most versatile tools for site scanning. To search for 404 errors in this program, enter the site URL in the address bar of the program, then click "Start", go to "Response Codes", select "Filter", and then "Client Error (4xx)". To find out where the 404 error is coming from, click on the line of the desired URL and go to the "Inlinks" tab.
Netpeak Spider is a paid program for SEO analysis of a site that also finds 404 errors. To view non-existent pages, you need to enter the URL of the site, click "Start", go to the "Reports" tab, then to the "Summary" tab. Then select "Page Status" → "Broken" or "Server Response Code" → "404 Not Found".
What should I do with broken links?
It is necessary to remove or replace the internal broken links with working URLs. It is recommended to check the resource for broken internal links at least once a month.
After the external broken links have been collected, you should:
Select broken URLs from sites with an X (citation index) greater than 500. These sites have a good reputation and traffic, which means a lot of traffic. The primary task is to analyze broken links on these resources.
Set up redirects from these links to pages with moved or similar content using 301 redirects. This will allow the user to click on the broken link and get to the page specified in the redirect. This way, the visitor will see the material he expected and will not leave the site.
Some webmasters use 301 redirects from external broken links to the main page of the site, but this is not the best practice. For example, if a user expected to see a catalog of smartphones, but got to the main page of an online store, he will have to search for the desired category in the menu and filter the products. This may lead to the loss of the buyer. Therefore, it is recommended to use 301 redirects to the main page only in extreme cases.
Correction of error 404
Understanding the principle of detecting problematic pages is only the first step. But how can I fix the 404 Page not found error? It all depends on the specific situation and the CMS used. Let's consider each case separately.
When using a CMS
Now many sites are created using CMS, such as WordPress, Joomla, 1C-Bitrix and others. If there are problems on such a site, and when you go to it you see a 404 error, you need to perform a number of actions to fix it.
First of all, go to the root folder of the site and create a file with the name there.htaccess. It should be a text file, the contents of which depend on the CMS used. You can find the standard directives on the official website reg.ru or in other useful materials on setting up htaccess.
However, sometimes the problem may remain even after the file is created.htaccess. In this case, you will have to contact the hosting technical support.
If the 404 error occurs due to the CMS itself
In such situations, you need to check each link for the correct input and make sure that the installed SEO plugins are functioning normally. After checking each link, the 404 error should disappear.
Without using a CMS
Sometimes sites function without using a CMS. In this case, the reason for the error may be due to the absence of files in the site folder or their incorrect location.
Be sure to check the root folder through the hosting panel and make sure that all the elements are present. In case of absence, upload them manually. If the error persists, contact the hosting technical support.
Now we have looked at what the 404 Not Found error is and how to fix it in various situations. The site owner should pay close attention to this issue and always be careful when compiling URLs. For ordinary users, it is important to enter the URL correctly in the browser's address bar.
How do I design a 404 page?
The main purpose of the 404 page is to inform the user that an error has occurred and indicate further actions. Despite the lack of clear design rules, there are four main approaches to design:
1. Entertainment option:
Mini-games, interesting facts or beautiful animation can be added here. It is important not to overload the content so that the user does not forget his original goal.
2. Advertising option:
Invite the user to familiarize themselves with your products or services and provide links for more information. You can also use special offers or issue promo codes to attract attention.
3. Search Page:
Add a search bar that allows users to search for information on your site or the entire Internet, extending the functionality of the 404 page.
4. Site map:
Place links to the main sections of your site so that the visitor can quickly find information of interest or learn about other useful sections.
How can I avoid problems with the 404 error?
Regularly check the site for broken links using programs such as Yandex.Webmaster, Google Search Console, Xenu's Link Sleuth, Screaming Frog SEO Spider Tool, Netpeak Spider. Make sure that all Not Found pages return the HTTP 404 response code. This way, search engines will understand that these addresses do not need to be scanned, and will focus on the really important URLs.
Do not leave the 404 page in its standard form — design it so as not to lose traffic. A well-designed page will keep users on the site and mitigate the negative from the detected error. The message on the page should clearly explain to the user what happened.
If you set 301 redirects from page 404, redirect only to pages with similar content. For example, to a page with a new version of the same product. Redirecting to irrelevant pages can mislead users: they expected to see one page, but ended up on another.
