The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

You are here: Home / Google / Google’s Googlebot Crawling, Search Visibility and Rankings

Google’s Googlebot Crawling, Search Visibility and Rankings

at PST By

When it comes to crawling the web, Googlebot crawls some pages with high frequency, with many crawls per day, while other pages can go months between Googlebot visits.  John Mueller from Google made some interesting comments regarding web pages that are crawled infrequently and why that might be, and if it is a concern for SEOs.

First, he says that regularly crawling is not a requirement for any infrequently crawled pages to rank or be visible in the search results.  So just because Googlebot doesn’t crawl a page daily, weekly or even monthly, that Google will exclude the page from the search results.

I suspect the crawling period is fine for things like that. Regular crawling isn’t a requirement for visibility or ranking in search.

However, the lack of Googlebot crawling certain URLs regularly can sometimes be a signal that there is a quality issue.  We know on pages with low quality links that it can take Googlebot some time to recrawl those URLs for disavow purposes. However, Googlebot can also crawl quality evergreen pages less frequently, especially when they are static pages with a history of not having any changes when Googlebot crawls.

However, in the case that John Mueller was responding to, the site owner stated that those infrequently crawled pages also get a small percentage of traffic, meaning they could be low quality or are pages Google is filtering out for being duplicates – something that is fairly common on ecommerce sites with near identical products.

(Often the URL count is also misleading, especially on ecomm a lot of those are often “unnecessary” URLs; filters/sorting/etc)

So if you have pages that are infrequently being crawled by Googlebot, it doesn’t mean Google won’t show them or rank them in their search results.  But it is advisable to see if there are underlying reasons that is causing Googlebot to crawl them less frequently, which could affect how well they do rank.

The following two tabs change content below.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.

Sign up for our newsletter