• About Us
  • Contributors
  • Guides
  • Speaking Engagements
  • Write for The SEM Post
  • Submit a tip or contact us!
  • Newsletters

The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

  • Google
  • SEO
  • Mobile
  • Local
  • Bing
  • Pay Per Click
  • Facebook
  • Twitter
  • State of the Industry
You are here: Home / Uncategorized / Google Slows Crawling With 5xx Server Errors

Google Slows Crawling With 5xx Server Errors

July 13, 2018 at 5:08 am PST By Jennifer Slegg

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS

When a website starts to serve Googlebot 5xx server errors, how does Googlebot handle the change in server status?  The question came up about Google crawl errors, and John Mueller from Google had some interesting comments about 5xx type of server errors and how it impacts Googlebot.

That's a very vague question, and a post from 2016, it makes it a bit hard to guess :). We do crawl old 404 pages from time to time, that's nothing to worry about. It's more problematic if they're 5xx errors (we slow crawling), or if the URLs were meant to be indexed.

— John ☆.o(≧▽≦)o.☆ (@JohnMu) July 9, 2018

So Googlebot will slow crawling once it begins to see 5xx server errors being returned.

For many sites, server errors tend to be for only a short period of time, such as a host being down, or a CMS update being pushed, and there would not any long term issues.  This happens to many websites from time to time.

For other sites, serving server errors can be a chronic issue and are a sign something is going on with the site.  Sometimes a site is serving just Googlebot 5xx server errors due to a misconfigured site firewall for example.  Or a site blocks traffic from certain countries, including ones Googlebot can crawl from, due to errors implementing geotargeting.

Regardless, if Search Console shows server errors, you should investigate them to make sure it was only a temporary problem and not a bigger issue. If Search COnsole continues to show errors, you should do some fetch and renders, as that will show exactly what Googlebot is seeing, and can help you investigate the errors if there doesn’t seem to be a problem when you visit the site.

Unless the server errors were a long term issue, Google should continue to visit the site to see if the server errors have been resolved and return to crawling as usual.  The crawl rate should go back to what the site usually sees after Googlebot is no longer seeing the errors.

 

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS
The following two tabs change content below.
  • Bio
  • Latest Posts
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Latest posts by Jennifer Slegg (see all)

  • 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates - August 1, 2022
  • Google Quality Rater Guidelines: The Low Quality 2021 Update - October 19, 2021
  • Rethinking Affiliate Sites With Google’s Product Review Update - April 23, 2021
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
  • Google Updates Experiment Statistics for Quality Raters - October 6, 2020

Filed Under: Google, SEO, Uncategorized

Sign up for our newsletter


Founder & Editor

Jennifer Slegg (2052)

Sign up for our daily news recap & weekly newsletter.


Follow us online

  • Facebook
  • Google+
  • Linkedin
  • Pinterest
  • Twitter

Latest News

2022 Update for Google Quality Rater Guidelines – Big YMYL Updates

We finally have the first Google Quality Rater Guidelines update of 2022, and like usual, it is … [Read More...]

Recent Posts

  • 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates
  • Google Quality Rater Guidelines: The Low Quality 2021 Update
  • Rethinking Affiliate Sites With Google’s Product Review Update
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met
  • Google Updates Experiment Statistics for Quality Raters
  • Analyzing “How Google Search Works” Changes from Google
  • Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations
  • Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites
  • Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags
  • Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials

Categories

  • Affiliate Marketing
  • Amazon
  • Apple
  • Bing
  • Branding
  • Browsers
  • Chrome
  • Content Marketing
  • Design
  • Domains
  • DuckDuckGo
  • Email
  • Facebook
  • Firefox
  • Foursquare
  • Google
    • Analytics
    • Google RankBrain
    • Quality Rater's Guidelines
  • History of Search
  • Industry Spotlight
  • Instagram
  • Internet Explorer
  • Links
  • Local
  • Mobile
  • Native Advertising
  • Other Search Engines
  • Pay Per Click
  • Pinterest
  • Publishers
  • Security
  • SEO
  • Snapchat
  • Social Media
  • State of the Industry
  • The SEM Post
  • Tools
  • Twitter
  • Uncategorized
  • User Experience
  • Video Marketing
  • Week in Review
  • Whitepapers
  • Wordpress
  • Yahoo
  • Yelp
  • YouTube
March 2023
MTWTFSS
« Aug  
 12345
6789101112
13141516171819
20212223242526
2728293031 

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Copyright © 2023 · News Pro Theme On Genesis Framework · WordPress · Log in