In yesterday’s Google Q&A with John Mueller and Andrey Lipattsev, I asked a question about Googlebot suddenly increasing the amount of crawling it does on a site, when there are no obvious reasons – such as a new URL structure – for Google to do so. Is it a sign of a Google update or an issue with the site? Or is it something that a site owner can’t really read anything into.
Some have claimed that increase crawl rate is directly related to an upcoming Penguin, Panda or other algo update. Some claim it is a sign that Google is evaluating a site for a possible manual action or being hit with an algorithmic issue by Google. But is this really the case?
First, Andrey Lippattsev joked and asked John Mueller if he would like to talk about the secret crawler that goes to a website if it has been naughty, which John responded “there is none.”
This is totally unrelated. So, the crawling that you might see is not related with manual actions, definitely not with manual actions, that’s something that is completely separate from crawling and generally not related to any other kind of algorithmic changes with a web site.
And I guess some of the reasons why we might crawl more are things that sometimes don’t even have much to do with the website, where our algorithms just decide oh, I want to double check all those URLs that I found on that website a while back and it just goes off and does that and suddenly we’re crawling twice as much for a couple of days and it doesn’t really mean that anything is a problem or anything that you need to worry about.
It’s essentially just our algorithms deciding that maybe today is a good day to look at this awesome website or maybe today is a good day to look at all these old URLs that we’ve found maybe 3 or 4 years ago and that we never got any content for and maybe there’s good content there today. So that’s something that’s not necessarily related to anything on the site or happening to the site.
So next time you see a spike in the number of pages being crawled on your site when there isn’t a specific reason why, it isn’t a sign that your site has been bad and is about to get hit with a manual action or some other Google search algo.
Jennifer Slegg
Latest posts by Jennifer Slegg (see all)
- Rethinking Affiliate Sites With Google’s Product Review Update - April 23, 2021
- New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
- Google Updates Experiment Statistics for Quality Raters - October 6, 2020
- Analyzing “How Google Search Works” Changes from Google - July 8, 2020
- Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations - December 6, 2019