GoogleBot Not Indexing Right Pages? Google Offers 5 Ways to Fix
Google’s John Mueller answered a Reddit question about indexing issues. Although the question was specific to an AngularJS issue, the solutions Google’s John Mueller offered are useful for all sites. His answer could be applied to almost anyone who asks, “Why isn’t Google indexing my content properly?”
Google’s Mueller offered five tips to diagnose canonicalization problem.
- Make Sure GoogleBot Can Index the Code
- Measure Content Processing Speed
- Minimize Resources Needed to Load Main Content
- Make it Easy to Reach Unique Content
- Canonicalization Issues May be a Symptom of Indexing Issues
1. Make Sure GoogleBot Can Index the Code
Here is a shortcut test John Mueller suggested to test if GoogleBot can index your site:
“If you’re sure that your code works for Googlebot (a quick way to test is the mobile-friendly test)…”
John Mueller suggested using Google’s Mobile Friendly Test as a way to test if GoogleBot can understand your code.
2. Content Processing Speed
Google’s John Mueller then encouraged web publishers to check if the amount of time necessary to process a page of content might be causing GoogleBot to improperly index content.
Here is what John Mueller said:
“…then it’s worth estimating if speed of processing might be an issue. The hard part here is that there’s no absolute guideline or hard cut-off point that you can focus on or test for, partially also because a page rarely loads exactly the same way across separate tests.
My way of eyeballing it is to see how long the mobile-friendly test roughly takes
and to check with webpagetest.org to see how long a page might take to load the critical / unique content, and how many resources are required to get there.”
3. Minimize Resources
Sites that load up fast convert more often, generate more sales and serve up more ad impressions. In order to speed up a site, review all the cookies and scripts needed to make that page load.
See Anyone’s Analytics Account, in Real Time.
You can literally see real-time sales and conversion data for any website, and which campaigns drove that traffic. Start your free trial today.
According to Google’s John Mueller:
“The more resources required, the longer time until the critical content is visible, the more likely Google will have trouble indexing the critical content.”
4. Make it Easy to Reach Content
GoogleBot is rendering content as if it’s a mobile browser (Chrome 41). Another consideration is something like chained redirects (a URL that redirects to another URL that subsequently redirects to another URL).
Test your site to make sure it renders well in the most popular Apple and Android devices. Doing things like blocking JS and CSS could cause GoogleBot to not render your pages well.
John Mueller said that content that is difficult to reach will result in Google being unable to properly index the content:
“…it’s probably a sign that it’s too hard to get to your unique content — it takes too many requests to load, the responses (in sum across the required requests) take too long to get back, so the focus stays on the boilerplate HTML rather than the JS-loaded content. Reducing dependencies & latency can help there.”
5. Canonicalization Issues May Be a Crawling Issue
The question on Reddit that John Mueller was responding to was about canonicalization issues. The person was asking what could he do to make Google canonicalize the right content.
“I’m having issues with canonicals on my site and I believe it’s due to the angular JS used in the HTML. For thousands of pages on the site Google is ignoring the canonical link and selecting their own…”
If you find that Google is canonicalizing and ranking the wrong page, it could be a symptom of an indexing problem. It might be that Google is having a difficult time rendering the web page. It could be too many resources are needed to load the page. It could also be that some resources are being blocked from seeing the unique content.
- Make it easy to render the site
- Make sure the site renders on Chrome 41
- Use Google’s Mobile Friendly test to gauge how long it takes to load
- Minimize resources need to load a web page
- Make it easy to get to the content