• About Us
  • Contributors
  • Guides
  • Speaking Engagements
  • Write for The SEM Post
  • Submit a tip or contact us!
  • Newsletters

The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

  • Google
  • SEO
  • Mobile
  • Local
  • Bing
  • Pay Per Click
  • Facebook
  • Twitter
  • State of the Industry
You are here: Home / Apple / The Apple Search Bot is Actually a Crawler to Improve Siri Not New Search Engine

The Apple Search Bot is Actually a Crawler to Improve Siri Not New Search Engine

November 12, 2014 at 5:54 am PST By Jennifer Slegg

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS

apple crawler
Everyone was buzzing earlier this week with rumors that Apple was crawling the web.    It began popping up in the middle of October with Apple’s IP range with a NetName of APPLE-WWNET as “Mozilla/5.0 (compatible; Fetcher/0.1)” written in Go.

The crawler does request robots.txt and is still actively crawling.  But the crawler is only taking straight HTML and not the CSS, JavaScript or image files.  While many crawlers don’t download CSS or JavaScript, the fact it was not indexing image files was interesting.

Of course, it had everyone speculating over the possibility that Apple might be getting into the search engine game, despite their current partnerships to provide search for iOS and Siri.  With Apple’s following and iOS default search, Apple could definitely make some inroads into creating their own search engine.

Here was the example log file from Jan Moesen.

    17.147.18.35 - - [06/Nov/2014:08:10:29 +0100] "GET /robots.txt HTTP/1.1" 301 185 "-" "Mozilla/5.0 (compatible; Fetcher/0.1)"
    17.147.18.35 - - [06/Nov/2014:08:10:30 +0100] "GET /robots.txt HTTP/1.1" 200 403 "http://www.catenacycling.com/robots.txt" "Go 1.1 package http"
    17.147.18.35 - - [06/Nov/2014:08:10:36 +0100] "GET / HTTP/1.1" 301 185 "-" "Mozilla/5.0 (compatible; Fetcher/0.1)"
    17.147.18.35 - - [06/Nov/2014:08:10:40 +0100] "GET /en HTTP/1.1" 200 93350 "http://www.catenacycling.com" "Go 1.1 package http"
    17.147.18.35 - - [06/Nov/2014:08:11:03 +0100] "GET /robots.txt HTTP/1.1" 200 403 "-" "Mozilla/5.0 (compatible; Fetcher/0.1)"
    17.147.18.35 - - [06/Nov/2014:08:11:05 +0100] "GET /en/ride-the-world/routes HTTP/1.1" 200 6379 "-" "Mozilla/5.0 (compatible; Fetcher/0.1)"
    17.147.18.35 - - [06/Nov/2014:08:11:09 +0100] "GET /en/ride-the-world/climbs HTTP/1.1" 200 6841 "-" "Mozilla/5.0 (compatible; Fetcher/0.1)"
    17.147.18.35 - - [06/Nov/2014:08:11:14 +0100] "GET /en/calendar/events HTTP/1.1" 200 9333 "-" "Mozilla/5.0 (compatible; Fetcher/0.1)"

But it would appear that the reality isn’t nearly as sexy as the possibility that Apple is laying down groundwork for a brand new search engine.  It is actually used by data scientists using statistical analysis and machine learning to improve Siri’s accuracy and performance.

@janmoesen apparently crawer is to improve siri: Using statistical analysis and machine learning to improve performance nd accuracy of Siri

— Kevin Deamandel (@DeamandelK) November 7, 2014

@janmoesen found a bug report mentioning the IP, checked Reporter's profile and saw he was a data scientist at apple mentioning siri.

— Kevin Deamandel (@DeamandelK) November 7, 2014

So while we may see Apple one day jump into the search engine game, this crawler isn’t a part of it at this time.

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS
The following two tabs change content below.
  • Bio
  • Latest Posts
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Latest posts by Jennifer Slegg (see all)

  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
  • Google Updates Experiment Statistics for Quality Raters - October 6, 2020
  • Analyzing “How Google Search Works” Changes from Google - July 8, 2020
  • Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations - December 6, 2019
  • Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites - September 13, 2019

Filed Under: Apple

Sign up for our newsletter


Comments

  1. Andrew Shotland says

    November 12, 2014 at 9:52 am

    I disagree with your conclusion. To me “using statistical analysis and machine learning to improve SIRI” does not imply Apple is not building a search engine. SIRI is a search engine.

Founder & Editor

Jennifer Slegg (2049)

Sign up for our daily news recap & weekly newsletter.


Follow us online

  • Facebook
  • Google+
  • Linkedin
  • Pinterest
  • Twitter

Latest News

New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met

Google has released a brand-new Google quality rater guidelines, coming close to a year after the … [Read More...]

Recent Posts

  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met
  • Google Updates Experiment Statistics for Quality Raters
  • Analyzing “How Google Search Works” Changes from Google
  • Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations
  • Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites
  • Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags
  • Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials
  • Google Local Service Ads Display Pricing Estimates for Specific Locations
  • Google Testing “Relevant History” Section in Mobile Search Results
  • Google Converts PDFs, DOCs, XLS etc into HTML for Indexing

Categories

  • Affiliate Marketing
  • Amazon
  • Apple
  • Bing
  • Branding
  • Browsers
  • Chrome
  • Content Marketing
  • Design
  • Domains
  • DuckDuckGo
  • Email
  • Facebook
  • Firefox
  • Foursquare
  • Google
    • Analytics
    • Google RankBrain
    • Quality Rater's Guidelines
  • History of Search
  • Industry Spotlight
  • Instagram
  • Internet Explorer
  • Links
  • Local
  • Mobile
  • Native Advertising
  • Other Search Engines
  • Pay Per Click
  • Pinterest
  • Publishers
  • Security
  • SEO
  • Snapchat
  • Social Media
  • State of the Industry
  • The SEM Post
  • Tools
  • Twitter
  • Uncategorized
  • User Experience
  • Video Marketing
  • Week in Review
  • Whitepapers
  • Wordpress
  • Yahoo
  • Yelp
  • YouTube
January 2021
MTWTFSS
« Oct  
 123
45678910
11121314151617
18192021222324
25262728293031

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Copyright © 2021 · News Pro Theme On Genesis Framework · WordPress · Log in