During the State of Search keynote yesterday in Dallas, Texas, Gary Illyes shared a pretty incredible stat – Google knows about 120 trillion URLs, and of those URLs, 60% of them are duplicate.
Now keep in mind Illyes does not say these are all indexed and ranking – merely that 60% of the URLs Google knows about are duplicate. But this is still an incredible number – more than half of them are duplicate!
We see Google regularly kick duplicates out of the search results – completely if it is a case of hardcore spam – or at least to not show them until someone clicks on the link at the bottom of the search results to reveal duplicates. There are also duplicates that get removed via a DMCA filed with Google, to remove stolen content from the search results.
Google doesn’t tend to release crawl stats very often, so it is nice to hear that there are 140 trillion URLs that Google knows about, as well as the fact 60% of those known URLs are duplicate.
Google knows about 120 trillion URLs, and 60% of them are duplicates o.O @methode #StateofSearch
— Jennifer Slegg (@jenstar) November 16, 2015
Jennifer Slegg
Latest posts by Jennifer Slegg (see all)
- Use Google Disavow Like Before, No Changes With Real Time Penguin - September 26, 2016
- No Changes to Link Related Manual Actions with Penguin Real Time Launch - September 26, 2016
- Google Launches Real-Time Penguin 4.0: All The Changes & Details - September 23, 2016
- Google Testing Individual Hotel Booking Block at Top of Search Results - September 22, 2016
- Recovering From Accidental Site Removal via Google Search Console Remove URL Tool - September 21, 2016
Reuben Yau says
I wonder if they also classify unintentional duplicates, like URLs with utm params, as part of that 60%.