If you have ever made a mistake with your robots.txt file – either by letting Googlebot crawl something you didn’t want it to or because you accidentally blocked Googlebot (or someone hacked your site and did it) – it can be some nail biting times until you see Googlebot picking up the new robots.txt changes.
Google has added an addition to their robots.txt tool. Now, you can do the usual check of your robots.txt file, but then you can also verify the live version and then submit it to notify Google that your robots.txt file has changed and request that Googlebot crawl it again.
You can access the new tool within Google Webmaster Tools by selecting “Crawl” in the left sidebar, and then choosing robots.txt Tester.
Latest posts by Jennifer Slegg (see all)
- Google Local Service Ads Display Pricing Estimates for Specific Locations - August 31, 2018
- Google Testing “Relevant History” Section in Mobile Search Results - August 31, 2018
- Google Converts PDFs, DOCs, XLS etc into HTML for Indexing - August 30, 2018
- Why Google Shows Featured Snippets With Images from Another Site - August 29, 2018
- How to Download Links Report in New Google Search Console - August 27, 2018