If you have ever made a mistake with your robots.txt file – either by letting Googlebot crawl something you didn’t want it to or because you accidentally blocked Googlebot (or someone hacked your site and did it) – it can be some nail biting times until you see Googlebot picking up the new robots.txt changes.
Google has added an addition to their robots.txt tool. Now, you can do the usual check of your robots.txt file, but then you can also verify the live version and then submit it to notify Google that your robots.txt file has changed and request that Googlebot crawl it again.
You can access the new tool within Google Webmaster Tools by selecting “Crawl” in the left sidebar, and then choosing robots.txt Tester.
Latest posts by Jennifer Slegg (see all)
- Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites - September 13, 2019
- Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags - September 10, 2019
- Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials - May 17, 2019
- Google Local Service Ads Display Pricing Estimates for Specific Locations - August 31, 2018
- Google Testing “Relevant History” Section in Mobile Search Results - August 31, 2018