Crawl Frequency on NoIndex & Unavailable Pages Decline Over Time

During Gary Illyes keynote with Stone Temple one of the questions was regarding how Google handles a long term noindex on a page in regards to Googlebot crawling that page, if the crawl frequency would decline over time.

Gary Illyes said that yes, they do drop the crawl frequency, not only for noindex, but for other problems with page availability as well.

Yes.  Typically it will decline for any page we cannot index for whatever reason.  Basically, we will try to crawl a few more times to see if the noindex is gone or if the page recovered from a 500 or whatever, and if the noindex is still there, then we will slowly start to move or to not crawl that page that often.

We will still probe that page every now and then, probably every two months or every three months, we will visit the page again to see if the noindex is still there. But we will very likely not crawl it that often anymore.

So the “for whatever reason” could also include cases where there are other long term issues on pages within a site, such as a database connection error, or other server responses such as Forbidden or 500 level server errors, not just pages that are noindexed.

Did you have a page with a noindex or was otherwise inaccessible for some time and are struggling to get it reindexed?  Your best bet is to do a fetch and submit in Google Search Console to attempt to jump start the crawling and indexing of the page.

The following two tabs change content below.

• Bio
• Latest Posts

Jennifer Slegg

Founder & Editor at The SEM Post

Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.

Latest posts by Jennifer Slegg (see all)

• Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites - September 13, 2019
• Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags - September 10, 2019
• Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials - May 17, 2019
• Google Local Service Ads Display Pricing Estimates for Specific Locations - August 31, 2018
• Google Testing “Relevant History” Section in Mobile Search Results - August 31, 2018