• About Us
  • Contributors
  • Guides
  • Speaking Engagements
  • Write for The SEM Post
  • Submit a tip or contact us!
  • Newsletters

The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

  • Google
  • SEO
  • Mobile
  • Local
  • Bing
  • Pay Per Click
  • Facebook
  • Twitter
  • State of the Industry
You are here: Home / Google / Google Uses Machine Learning for Crawling, Indexing & Ranking

Google Uses Machine Learning for Crawling, Indexing & Ranking

May 2, 2018 at 7:20 am PST By Jennifer Slegg

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS

There has been plenty of speculation about just how much Google uses machine learning when it comes to the search algorithm, especially outside of RankBrain itself.  During a recent Google Webmaster Office Hours, the role of machine learning in the algo came up.

The question was specifically about Panda and whether it uses a machine learning algo classifier on sample pages to train high quality versus low quality content.

John Mueller doesn’t specifically talk about Panda and machine learning, but instead he discusses Google’s use of machine learning overall in their search algos.  And while we know Google does use machine learning for RankBrain, which enables Google to serve high quality results for the type of search queries that are never or rarely seen, they haven’t talked much about it from outside of RankBrain.

First, Mueller talks about the Panda aspect of it.

I don’t actually know what the Panda algorithm  uses so I can’t really answer that question for you.

He does clarify later that it wouldn’t be out of the question for Google Panda to have a machine learning component, but he doesn’t know.

Then he goes on to talk about how Google is using machine learning, and he says Google is using it to understand better “how we should crawl, index and rank pages.”

In general, we do use a lot of machine learning to try and better understand how we should crawl, index and rank pages, so that’s something that wouldn’t be totally out of the question, but as far as I know, or actually I don’t know what the Panda algorithm does there.

So that’s kind of something where I think machine learning has a lot of potential to try and understand pages a little bit better.  It’s not like an automatic solution, though.   It’s not that we can just like feed it a bunch of pages and say these are good, these are bad and then it’ll figure out the whole rest of the web on its own.  It does take quite a bit of work to actually make machine learning work well enough.

Ever since Google first announced RankBrain, and the use of machine learning in algo, many SEOs have been speculating about just how much it is used, especially beyond the scope of RankBrain.

The use of machine learning to help determine crawl budget could mean that what a larger scope of sites do could impact an individual site’s crawl budget as well, despite the fact many SEOs manipulate how large sites are crawled, such as through robots.txt, noindex and nofollow, to see their site crawled optimally.  But if Google is using machine learning, whether for training or directly, to help determine how they should crawl a site, SEO’s own crawl budgeting could have less of an impact, especially in the future as we know Google use using machine learning more and more.

As for indexing, it is harder to know for certain how Google is using it, or again if it is merely for training purposes.

Here are the full comments:

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS
The following two tabs change content below.
  • Bio
  • Latest Posts
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Latest posts by Jennifer Slegg (see all)

  • 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates - August 1, 2022
  • Google Quality Rater Guidelines: The Low Quality 2021 Update - October 19, 2021
  • Rethinking Affiliate Sites With Google’s Product Review Update - April 23, 2021
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
  • Google Updates Experiment Statistics for Quality Raters - October 6, 2020

Filed Under: Google, SEO

Sign up for our newsletter


Founder & Editor

Jennifer Slegg (2052)

Sign up for our daily news recap & weekly newsletter.


Follow us online

  • Facebook
  • Google+
  • Linkedin
  • Pinterest
  • Twitter

Latest News

2022 Update for Google Quality Rater Guidelines – Big YMYL Updates

We finally have the first Google Quality Rater Guidelines update of 2022, and like usual, it is … [Read More...]

Recent Posts

  • 2022 Update for Google Quality Rater Guidelines – Big YMYL Updates
  • Google Quality Rater Guidelines: The Low Quality 2021 Update
  • Rethinking Affiliate Sites With Google’s Product Review Update
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met
  • Google Updates Experiment Statistics for Quality Raters
  • Analyzing “How Google Search Works” Changes from Google
  • Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations
  • Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites
  • Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags
  • Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials

Categories

  • Affiliate Marketing
  • Amazon
  • Apple
  • Bing
  • Branding
  • Browsers
  • Chrome
  • Content Marketing
  • Design
  • Domains
  • DuckDuckGo
  • Email
  • Facebook
  • Firefox
  • Foursquare
  • Google
    • Analytics
    • Google RankBrain
    • Quality Rater's Guidelines
  • History of Search
  • Industry Spotlight
  • Instagram
  • Internet Explorer
  • Links
  • Local
  • Mobile
  • Native Advertising
  • Other Search Engines
  • Pay Per Click
  • Pinterest
  • Publishers
  • Security
  • SEO
  • Snapchat
  • Social Media
  • State of the Industry
  • The SEM Post
  • Tools
  • Twitter
  • Uncategorized
  • User Experience
  • Video Marketing
  • Week in Review
  • Whitepapers
  • Wordpress
  • Yahoo
  • Yelp
  • YouTube
December 2025
MTWTFSS
« Aug  
1234567
891011121314
15161718192021
22232425262728
293031 

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Copyright © 2025 · News Pro Theme On Genesis Framework · WordPress · Log in