• About Us
  • Contributors
  • Guides
  • Speaking Engagements
  • Write for The SEM Post
  • Submit a tip or contact us!
  • Newsletters

The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

  • Google
  • SEO
  • Mobile
  • Local
  • Bing
  • Pay Per Click
  • Facebook
  • Twitter
  • State of the Industry
You are here: Home / Google / How to Unblock All CSS & Javascript for Googlebot Using Robots.txt

How to Unblock All CSS & Javascript for Googlebot Using Robots.txt

July 29, 2015 at 3:19 am PST By Jennifer Slegg

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS

how to unblock css jsIf you have received one of the many warnings Google sent out about “Googlebot cannot access CSS and JS files” in your Google Search Console, there has been much confusion about how to ensure you are allowing Googlebot to crawl these blocked resources without potentially opening yourself up to security issues, as the commonly blocked parts of WordPress – /wp-admin/ and /wp-content/plugins – are triggering the warnings, depending on how WordPress and various plugins are set up.

Fortunately, there is an easy way to allow Googlebot to crawl all CSS and JavaScript files while still allowing other areas to remain blocked.

Simply add the following to your robots.txt and it will allow Googlebot to crawl all the CSS and JavaScript files on your site.

User-Agent: Googlebot
Allow: .js
Allow: .css

Also remember that Google does not index .css and .js in their search results, so you do not need to worry about these files causing any unnecessary SEO issues.  In fact, the opposite will happen since blocking .css and .js files “can result in suboptimal rankings.”

Once you add this, you can confirm using the Google Search Console “Fetch as Google” to ensure the resources were successfully unblocked with the new robots.txt file.

While some commenters pointed out a similar robots.txt in comments here, Gary Illyes from Google confirmed the use of this in robots.txt at stackoverflow.

…the simplest form of allow rule to allow crawling javascript and css resources:

User-Agent: Googlebot
Allow: .js
Allow: .css

This will allow anything like https://example.com/deep/style.css?something=1 or https://example.com/deep/javascript.js, and leaves no much space for interpretation for other search engines.

Once you have this, you can test your setup in Search Console – Blocked resources feature.

We have more details on the “Googlebot cannot access CSS and JS files” warning here, and here are instructions so you can find exactly which resources Google thinks you are blocking, even if you are certain it is a false positive (it probably isn’t).

  • Facebook
  • Twitter
  • Google+
  • Pinterest
  • LinkedIn
  • Email
  • WhatsApp
  • Evernote
  • SMS
The following two tabs change content below.
  • Bio
  • Latest Posts
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Latest posts by Jennifer Slegg (see all)

  • Google Quality Rater Guidelines: The Low Quality 2021 Update - October 19, 2021
  • Rethinking Affiliate Sites With Google’s Product Review Update - April 23, 2021
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met - October 16, 2020
  • Google Updates Experiment Statistics for Quality Raters - October 6, 2020
  • Analyzing “How Google Search Works” Changes from Google - July 8, 2020

Filed Under: Google, SEO

Sign up for our newsletter


Comments

  1. Arun says

    July 30, 2015 at 12:16 am

    Yes, this is mandate now and it seems google has revamped its algorithm of ranking with the JS and CSS usability as prime factor.

    I have received for all most all the clients having the wordpress where i blocked the access to wp-include through robots file.

  2. Sandip Paul says

    July 30, 2015 at 12:36 am

    Thanks a lot. This help a lot …

  3. Uzma Shahzadi says

    July 30, 2015 at 1:11 am

    I think the solution is incomplete. Directives mentioned in this article are not complete. Only “Allow: .js” will allow only those “.js” file that are at root level.

    To allow all the “.js” and “.css” files, the wild card operator (*) is required. See the code below;

    User-Agent: Googlebot
    Allow: *.js
    Allow: *.css

    • Ahmad Ali says

      July 31, 2015 at 3:59 am

      Uzma that’s really good answer I agree with you.

    • Casey Markee says

      August 4, 2015 at 12:53 pm

      Yes, the solution referenced in this article is INCORRECT. You must have the “wildcard” command in the lines for this to work.

      This is correct:

      User-Agent: Googlebot
      Allow: *.js
      Allow: *.css

      This is NOT correct:

      User-Agent: Googlebot
      Allow: .js
      Allow: .css

      Make sure to include this at the top of the robots file since commands are read from top to bottom.

      Yoast has published a great new resource stating that you should also be removing (unblocking) access to /wp-admin/ as well.

      That’s a must-read and you can find that resource here (read the comments as well).

      https://yoast.com/dont-block-your-css-and-js-files/

  4. Andrea says

    July 30, 2015 at 4:55 am

    The suggestion is definitely wrong cause it doesn’t cover every case.

    Apart from that suggesting a predefined robots.txt is never a good idea.

    Never disallow anything except if you know what you are doing and after looking your server log files.

  5. johnsmith007 says

    August 5, 2015 at 9:51 pm

    If you have wordpress website then should need to edit like first go your wordpress dashboard
    and choose seo section and click on edit files and

    default code like

    User-agent: *
    Disallow: /wp-admin/

    insert new text below this code like

    User-Agent: Googlebot
    Allow: *.js
    Allow: *.css

Trackbacks

  1. More From Google on Blocked CSS & JavaScript Warnings says:
    July 29, 2015 at 4:45 am

    […] If you want the easy way to unblock your JavaScript and CSS, here is how to do it. […]

Founder & Editor

Jennifer Slegg (2051)

Sign up for our daily news recap & weekly newsletter.


Follow us online

  • Facebook
  • Google+
  • Linkedin
  • Pinterest
  • Twitter

Latest News

Google Quality Rater Guidelines: The Low Quality 2021 Update

Google has released a new version of the Google quality rater guidelines, a year after the last … [Read More...]

Recent Posts

  • Google Quality Rater Guidelines: The Low Quality 2021 Update
  • Rethinking Affiliate Sites With Google’s Product Review Update
  • New Google Quality Rater Guidelines, Update Adds Emphasis on Needs Met
  • Google Updates Experiment Statistics for Quality Raters
  • Analyzing “How Google Search Works” Changes from Google
  • Google Quality Rater Guidelines Update: New Introduction, Rater Bias & Political Affiliations
  • Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites
  • Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags
  • Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials
  • Google Local Service Ads Display Pricing Estimates for Specific Locations

Categories

  • Affiliate Marketing
  • Amazon
  • Apple
  • Bing
  • Branding
  • Browsers
  • Chrome
  • Content Marketing
  • Design
  • Domains
  • DuckDuckGo
  • Email
  • Facebook
  • Firefox
  • Foursquare
  • Google
    • Analytics
    • Google RankBrain
    • Quality Rater's Guidelines
  • History of Search
  • Industry Spotlight
  • Instagram
  • Internet Explorer
  • Links
  • Local
  • Mobile
  • Native Advertising
  • Other Search Engines
  • Pay Per Click
  • Pinterest
  • Publishers
  • Security
  • SEO
  • Snapchat
  • Social Media
  • State of the Industry
  • The SEM Post
  • Tools
  • Twitter
  • Uncategorized
  • User Experience
  • Video Marketing
  • Week in Review
  • Whitepapers
  • Wordpress
  • Yahoo
  • Yelp
  • YouTube
July 2022
MTWTFSS
« Oct  
 123
45678910
11121314151617
18192021222324
25262728293031

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Copyright © 2022 · News Pro Theme On Genesis Framework · WordPress · Log in