• About Us
  • Contributors
  • Guides
  • Speaking Engagements
  • Write for The SEM Post
  • Submit a tip or contact us!
  • Newsletters

The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

  • Google
  • SEO
  • Mobile
  • Local
  • Bing
  • Pay Per Click
  • Facebook
  • Twitter
  • State of the Industry
You are here: Home / Google / Googlebot Behavior When Major Changes Made to Site

Googlebot Behavior When Major Changes Made to Site

March 23, 2018 at 5:43 am PST By Jennifer Slegg

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS

The question about Googlebot being allowed to crawl pages that had previously been blocked from Googlebot. Googlebot crawl activity, as shown in Google Search Console crawl stats, showed a huge spike one day, which doubled the next day and then tripled the third.

But Google wasn’t necessarily hitting the “new to Googlebot” pages, but crawling old cache pages as well.  John Mueller, in the last Google Webmaster Office Hours discussed in detail how Googlebot handles crawling when it recognizes major changes to a site.

I don’t think that would be – so I’m not exactly sure where you mean or what that looks like, but it doesn’t sound like that would be something like an effect that you would see from Google’s crawling.  Because usually what would happen in a case like this is if we recognize that there are significant changes on the website, we’ll just try to crawl the known URLs a little bit faster.

So we have a bunch of URLs that we already know from your website, and we basically decide, “oh, we want to make sure that our index is as fresh as possible.  Therefore, we’ll take this list of URLs and crawl them as quickly as we can.  It kind of depends on your server, what we think your server can take.  But we’ll try to get through that a little bit faster than normal.

So that’s particularly the case where we find significant changes across a site with regards to maybe structured data or with the URL choices, with real canonical redirects, those kinds of things.  So maybe that’s something that was triggered from your side, depending on what you changed.

But in general, that would be visible like normal Googlebot requests.  We would just go through our list of URLs, and we just check them all.  It wouldn’t be the case that you would see something like a Chrome cache or something.  So I suspect what you are seeing there is either an effect that Analytics picked up kind of incorrectly.

Mueller also mentioned that it is possible the increased crawling and the changes are completely unrelated as well.  SEOs are familiar with Googlebot suddenly spiking with activity on a site, even if nothing has been changed, particularly nothing that would be deemed significant.

Maybe, or maybe it’s totally unrelated.  Maybe it’s just Googlebot was crawling and this other random thing was happening at the same time, and it looks like they’re related but actually, they are two completely separate things.

Of course, if the site is low quality, then it can still take quite some time for Googlebot to recrawl those low quality pages – sometimes six months or more before changes in all those pages can be reflected in the search results.

It is worth noting that increased crawling does not mean increased indexing or rankings.  So if you are concerned Googlebot is hitting your site too hard, especially if you expect Googlebot to hit your site hard, you can throttle the amount it crawls.  But Googlebot is usually pretty good at recognizing when it might be crawling harder than what the site can support.

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS
The following two tabs change content below.
  • Bio
  • Latest Posts
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Latest posts by Jennifer Slegg (see all)

  • 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates - August 1, 2022
  • Google Quality Rater Guidelines: The Low Quality 2021 Update - October 19, 2021
  • Rethinking Affiliate Sites With Google’s Product Review Update - April 23, 2021
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
  • Google Updates Experiment Statistics for Quality Raters - October 6, 2020

Filed Under: Google, SEO

Sign up for our newsletter


Founder & Editor

Jennifer Slegg (2052)

Sign up for our daily news recap & weekly newsletter.


Follow us online

  • Facebook
  • Google+
  • Linkedin
  • Pinterest
  • Twitter

Latest News

2022 Update for Google Quality Rater Guidelines – Big YMYL Updates

We finally have the first Google Quality Rater Guidelines update of 2022, and like usual, it is … [Read More...]

Recent Posts

  • 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates
  • Google Quality Rater Guidelines: The Low Quality 2021 Update
  • Rethinking Affiliate Sites With Google’s Product Review Update
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met
  • Google Updates Experiment Statistics for Quality Raters
  • Analyzing “How Google Search Works” Changes from Google
  • Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations
  • Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites
  • Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags
  • Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials

Categories

  • Affiliate Marketing
  • Amazon
  • Apple
  • Bing
  • Branding
  • Browsers
  • Chrome
  • Content Marketing
  • Design
  • Domains
  • DuckDuckGo
  • Email
  • Facebook
  • Firefox
  • Foursquare
  • Google
    • Analytics
    • Google RankBrain
    • Quality Rater's Guidelines
  • History of Search
  • Industry Spotlight
  • Instagram
  • Internet Explorer
  • Links
  • Local
  • Mobile
  • Native Advertising
  • Other Search Engines
  • Pay Per Click
  • Pinterest
  • Publishers
  • Security
  • SEO
  • Snapchat
  • Social Media
  • State of the Industry
  • The SEM Post
  • Tools
  • Twitter
  • Uncategorized
  • User Experience
  • Video Marketing
  • Week in Review
  • Whitepapers
  • Wordpress
  • Yahoo
  • Yelp
  • YouTube
March 2023
MTWTFSS
« Aug  
 12345
6789101112
13141516171819
20212223242526
2728293031 

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Copyright © 2023 · News Pro Theme On Genesis Framework · WordPress · Log in