The SEM Post

Latest News About SEO, SEM, PPC & Search Engines

You are here: Home / Google / Google’s DoubleClick Outage Affected 55,000 Websites For Nearly Two Hours

Google’s DoubleClick Outage Affected 55,000 Websites For Nearly Two Hours

at PST By

doubleclickpublishersWhen Google stopped serving DoubleClick Ads on publisher websites last Wednesday, it turns out that more than 55,000 publishers were affected.  The sites,  which included major websites such as Forbes.com, The Wall Street Journal and The Guardian, caused issues beyond simple ads failing to load, as it seemed to cause some websites to hang as they tried to connect to the DoubleClick servers for ads.

The outage, which lasted from 5:45 am through 7:31am PST, was the first major outage for DoubleClick’s DoubleClick for Publishers (DFP) program.

Google’s Neal Mohan, VP, Display & Video Advertising and Scott Silver, VP, Engineering sent an email to publishers explaining the cascading issue that caused the loss of ad serving.

The details:
• The DFP ad server relies on an internal service that began degrading in performance. This caused a cascading failure on DFP ad servers, leading to the outage.
• We designed our systems to gracefully handle performance degradation from dependent services. However, due to a misconfiguration, we were unable to prevent the outage.
• To restore ad serving and prevent cascading failures, we restarted the services by provisioning additional resources.
• We reproduced the failure in a test by degrading the availability of the internal service, proving the misconfiguration caused the cascading failures. We have since rolled out a fix to the configuration globally.
• We are conducting a complete review of all our processes and production configurations to prevent this from happening again.

It is unknown if Google will offer any compensation for those websites and publishers that were affected by the outage.

The following two tabs change content below.
My Twitter profileMy Facebook profileMy Google+ profileMy LinkedIn profile

Jennifer Slegg

Founder & Editor at The SEM Post
Jennifer Slegg is a longtime speaker and expert in search engine marketing, working in the industry for almost 20 years. When she isn't sitting at her desk writing and working, she can be found grabbing a latte at her local Starbucks or planning her next trip to Disneyland. She regularly speaks at Pubcon, SMX, State of Search, Brighton SEO and more, and has been presenting at conferences for over a decade.

Sign up for our newsletter