The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

You are here: Home / Google / Sites Can Have Different Crawl Frequency on a Per URL Basis

Sites Can Have Different Crawl Frequency on a Per URL Basis

at PST By

crawl frequencyJohn Mueller commented in his new post about reporting in Google Search Console that sites can have different crawl rates on a per-URL basis.  In other words, don’t expect that just because Google crawls your home page and few more important pages daily that Googlebot is hitting each individual page on the site daily too.

Google tends to crawl pages that it knows likely haven’t changed with much less frequency.  If it knows you added the page in 2008 and nothing has ever changed with the main content, and perhaps the only changes have been to the site’s overall navigation structure, don’t be surprised if Google only recrawls that page every few months – even if it is hitting other pages much more frequently or even daily.

This is also why when disavowing links it is recommended you disavow on the root domain, rather than individual URLs.  When Google does it’s daily crawl of the homepage, it will process your disavow then, while for some obscure internal page, Google could take months to crawl the individual internal URL and process that disavowed link.

So of course, this also applies to the errors that show up on individual URLs within Google Search Console.  Wile many people love to see their sites with no crawl errors at all, it can be frustrating to see them still showing up as errors despite being fixed.  You can also submit individual URLs for crawling, but this isn’t a feasible solution on a large scale.

Here is his full comment:

Per-URL crawl rates differ. Some URLs are crawled every few minutes, others just every couple months, and many somewhere in between. If you go from “all URLs are broken” to “all URLs are fixed” (which is awesome if you have a way to do that!), it will take some time to drop to “zero errors” in the aggregated reports. This is probably too much, but I created a simple spreadsheet to show that, feel free to make a copy & play with it: https://goo.gl/1pToL8 (screenshot here too)

The following two tabs change content below.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.

Sign up for our newsletter