Common Google Indexing Issues & How to Fix Them 

“Why isn’t my new piece of content showing up in Google’s search results?”

If you find yourself asking this question, it’s highly likely that an indexing error is the culprit. 

For a web page to appear in Google’s search results, it has to be crawled and indexed first, and this is a process that can take a while. 

According to Google’s John Mueller, indexing a new page can take anywhere from several hours to several weeks. So, if you recently published a piece of content that’s not indexed, it could be that Googlebot hasn’t had the time to crawl it yet. 

Also, there are tons of indexing errors that will stop your content from appearing in the search results. 

These range from Javascript rendering issues to improperly formatted robots.txt files, so there’s a lot to look out for. 

The good news is most indexing errors are relatively easy to fix if you know where to look. 

In this guide, we’ll teach you how to identify and resolve the most common Google indexing issues, so don’t go anywhere! 

Why is Indexing Important for Search Engines?

Before we dive into some common indexing errors, it’s important to know why the indexing process matters in the first place. 

In order for web pages to appear in Google’s search results, they have to show up in its index first, which is where the crawling process enters the picture. 

Crawling is where a bot (called a crawler) reads the content of a page and follows any links it finds in order to:

1. Understand the topics and themes explored by the web page/website. 
2. Locate keywords related to common search queries. 
3. Determine if the content is worthy of a spot in Google’s index.

Google uses automated software called Googlebot to crawl websites on the internet, and there are two specific types:

• Googlebot Smartphone is Google’s mobile crawler that simulates a user on a smart device. 

• Googlebot Desktop is a desktop crawler that acts as a user on a desktop PC. 

So, if your content is determined to be relevant and high-quality, it may appear in Google’s index. 

We chose the word may because indexing is never a guarantee. 

As John Mueller points out, it’s normal for 20% of a website not to be indexed, and there are many reasons why that is. 

Why might your content NOT appear in Google’s index?

Besides the common indexing errors that we’re about to explore, there are a few other reasons why pages on your site might not get indexed. 

These include:

• A noindex tag (this HTML tag tells Googlebot not to include the content in its index) 
• If the index already contains plenty of higher-quality content on the subject you covered 

Noindex tags come in handy for pages you don’t want to appear in Google’s index, such as admin pages or log-in pages. 

Since these pages won’t provide any value to your business (i.e., they won’t generate leads and sales), there’s no reason to include them in Google’s search results (or to try to get them to rank for any keywords). 

Also, if there’s plenty of high-quality content that already exists in Google’s index for a certain piece of content, it may not get indexed. 

Here’s a direct quote from John Mueller on the matter:

“With regards to the quality, when it comes to understanding the quality of a website, that is something that we take into account quite strongly with regards to crawling and indexing the rest of the website.”

So, if a few of your blog posts and videos aren’t getting indexed despite no technical issues, it’s likely a quality issue. 

In that case, your best bet is to reevaluate the quality of your content. If at all possible, explore new topics or try to provide insights on familiar topics that others ranked on Google haven’t provided yet.

How to Spot Indexing Errors on Your Website 

Before you can resolve an indexing error, you need to know that it exists first. 

The quickest and easiest way to discover if you have any indexing errors is to use Google Search Console (GSC). 

It’s a free tool from Google that lets you view your SEO as Google sees it (which is extremely helpful for lots of SEO-related tasks). 

If you aren’t set up on GSC yet, you’ll have to verify ownership of your website first. 

Here’s an in-depth guide on Google Search Console that will get you completely up to speed on the platform. 

To view any indexing errors you may have, log in to GSC and navigate to Pages under the Index category on the left-hand sidebar. 

This will take you to the Page Indexing Report. 

At the top of the page, you get to view how many of your web pages Google has indexed vs. not indexed, which is a handy way to eyeball how much of your website is indexed. 

If you keep scrolling down the page, you’ll see a section titled Why pages aren’t indexed, which will contain a complete list of any indexing errors you currently have. 

This report tells you:

1. The reason for the error (i.e., soft 404s, noindex tags, redirects, etc.).
2. The source of the error (most of the time, it’s your website, but that’s not always the case). 
3. Whether you’ve validated the fix yet or not. 
4. The indexing trend (how common the issue is). 
5. The number of pages affected by the error. 

Essentially, this report tells you everything you need to know about an error in order to fix it. 

All you have to do is click on one of the reasons for an error, and you’ll get to see a more detailed report that contains a short description of the problem. 

 

Clicking on Learn More will direct you to an appropriate section of GSC’s Help documents. 

Once you’ve resolved the issue, you can click the Validate Fix button to let Google know that the problem is fixed. 

 

As you can see in this example, 250 entire web pages aren’t getting indexed due to 404 Not Found errors. That’s a lot of broken links, but it’s not unheard of for a larger website to have that many (or more). 

The Most Common Indexing Issues and Their Fixes 

Now that you know how to compile a list of all your indexing errors (if you have any), let’s learn how to fix them. 

Here are the most common indexing errors and how to resolve them. 

Error type #1: 4xx errors 

First, let’s look at some 4xx errors you may run into, which indicate that a webpage doesn’t exist or has restricted access. 

There are a few types of 4xx errors that tend to pop up during crawling and indexing, so let’s look at them all. 

Unauthorized request (401)

If one of your URLs returns a 401 error code, it won’t be able to get crawled or indexed. The 401 error means that the requester isn’t allowed to access the web page. 

This means that the server understands what the request is, but it refuses to authorize it due to a lack of (or improper) credentials. 

To fix a 401, you should:

• Check if the website is down for everyone and not just you 
• Clear your cookies and cache 
• Check authentication credentials (like for a password-protected page) 
• Clear your DNS (domain name system) cache 
• Turn off password protection 
• Disable plugins and themes 

Access forbidden (403)

Similar to a 401, a 403 error code means that you’re trying to access a resource that you aren’t allowed to access. 

However, in the case of 403s, access is forbidden. 

Some of the same fixes for 401s can work for 403s, like disabling plugins and clearing your cookies and cache. 

You should also:

• Scan for malware 
• Reset directory and file permissions
• Disable CMS plugins 

Not found (404)

The most common 4xx error is the infamous 404 Not Found. 

As the name implies, the server couldn’t find the web page because it no longer exists. It could have been moved to a new URL, but no redirect has been implemented as of yet. 

The simplest way to fix a 404 Not Found is to add a 301 redirect to the new page (or a similar page if the old content was deleted). 

Error type #2: Redirect errors 

While we’re on the subject of redirects, they can wind up causing indexing errors of their own, too. 

There are a few ways redirect errors can occur, including the following:

1. The redirect loops back to the same page (in which case, you need to change it to a new URL). 
2. The redirect URL exceeded the maximum character length (shortening it will fix this). 
3. The redirect chain is too long (simplifying the redirect chain fixes this problem). 
4. There may be an empty or incorrect URL in a redirect chain (removing or correcting these is the way to go). 

So, if GSC notifies you of a redirect error, check for one of the above. 

Error type #3: URL issues 

If a URL is incorrect or blocked by your robots.txt file, Google won’t be able to crawl or index it. 

Common URL issues (and their fixes) include the following:

1. The URL has a noindex tag (removing the tag will solve the problem). 
2. The URL is blocked by your robots.txt file (removing the URL from the file fixes this. This guide contains more information about formatting robots.txt files). 
3. There’s an incorrect or empty URL. This is an easy fix, as you either have to A) correct the spelling of a URL or B) add a URL where one is missing. 

Error type #4: Content issues 

Your content can also cause the crawling process to flub up, such as:

1. 1. Duplicate content. This occurs whenever two identical pages are trying to rank for the same keyword. Duplicate content can appear on any website, but it’s most prevalent in e-commerce stores that have very similar pages for different product details (such as size and color). To resolve duplicate content, you should either delete the duplicate page or set the main page as canon with a canonical tag. 

• Thin content. Google views content with less than 700 – 1,000 words as thin, so it won’t bother indexing it. To remedy this issue, you should ensure each page you publish (and want indexed) has at least 1,000 words or more. 

• New content. If your content is super fresh (i.e., you just published it yesterday), it’s likely too new to appear in Google’s index. All you have to do for this problem is play the waiting game!

Error type #5: Site structure problems 

Sometimes the crawling process goes awry because Googlebot can’t make heads or tails out of your website’s structure. 

For the crawling and indexing process to go as smoothly as possible, you need a logical site and URL structure that features plenty of internal links. 

Common site structure issues include:

• Having too many inner pages. A flat site design works best for SEO, where each page is only a click or two away from the homepage. Otherwise, Googlebot may become lost in your intricate web of inner pages. 

• Poor mobile optimization. Google uses mobile-first indexing, so Googlebot is going to crawl the mobile version of your website first. If it doesn’t exist (or is improperly formatted), you’ll run into trouble. Check out our guide on mobile optimization to learn how to make your website as mobile-friendly as possible. 

• Orphan pages. An orphan page occurs whenever a web page has no internal links pointing at it. As such, Googlebot won’t be able to find it since it uses internal links to navigate your site during the crawling process. As a rule of thumb, always include at least one internal link on every web page on your site. You can also use an SEO spider like Screaming Frog to track down any existing orphan pages. 

• Missing sitemap. One of the best ways to ensure optimal crawlability for your site is to upload your sitemap to Google Search Console. It acts as a roadmap for Googlebot when crawling your site, so you should always upload it to GSC. 

Error type #6: You exceeded your crawl budget 

Sometimes web pages won’t get indexed because Googlebot has already used up its allotted crawl budget for your website. 

Crawling the internet takes a lot of energy and resources, which is why Google assigns a crawl budget to each web page to conserve power. 

The most popular websites online receive the biggest crawl budgets, with smaller websites receiving tinier budgets. 

To get the most out of your crawl budget, ensure that you’re only attempting to index your most important pages for SEO. Everything else should go in your robots.txt file or get hit with a noindex tag. 

Error type #7: Google penalty 

If you’ve tried all the fixes above and the web pages on your website STILL aren’t appearing in Google’s index, you may have a manual action on your website. 

Also called a manual penalty, this will cause your content to disappear from the SERPs until the issue is resolved. 

Common causes for penalties on Google include guideline violations like paid links, linking to malicious websites, and sneaky redirects. 

To check to see if you have a penalty, log into GSC and navigate to the Security and Manual Actions tab on the left-hand side. This report will let you know if you have any penalties against your site, as well as actions that you need to take to get rid of them (such as disavowing backlinks from a link farm). 

Get Rid of Your Indexing Errors for Better SEO 

Indexing errors aren’t something you want to deal with on a recurring basis, so let’s recap what we’ve covered so far:

1. For web pages to appear in Google’s search results, they must appear in its index first. 
2. Google uses Googlebot to crawl websites on the internet. 
3. Various factors can interrupt the crawling process and cause certain pages to not appear in the index. 
4. You can use Google Search Console to quickly identify all the crawling and indexing errors on your website. 
5. Fixing these errors as soon as they pop up will ensure your most important pages for SEO get indexed. 

Would you love some help resolving the technical SEO issues on your site (like indexing errors)?

During one of our Technical SEO Audits, our expert team will hunt down every single technical issue affecting your SEO and provide you with an incredibly detailed report and action plan. 

From there, you can either handle the fixes yourself, or you can hire our talented team to resolve all your issues! 

It’s a hassle-free way to handle technical SEO, which is a notoriously cumbersome process, so don’t wait to try it out. 

Also, don’t wait to reach out for a free SEO consultation to discuss your specific needs in more detail!    

The post Common Google Indexing Issues & How to Fix Them  appeared first on The HOTH.