When Google stopped serving DoubleClick Ads on publisher websites last Wednesday, it turns out that more than 55,000 publishers were affected. The sites, which included major websites such as Forbes.com, The Wall Street Journal and The Guardian, caused issues beyond simple ads failing to load, as it seemed to cause some websites to hang as they tried to connect to the DoubleClick servers for ads.
The outage, which lasted from 5:45 am through 7:31am PST, was the first major outage for DoubleClick’s DoubleClick for Publishers (DFP) program.
Google’s Neal Mohan, VP, Display & Video Advertising and Scott Silver, VP, Engineering sent an email to publishers explaining the cascading issue that caused the loss of ad serving.
• The DFP ad server relies on an internal service that began degrading in performance. This caused a cascading failure on DFP ad servers, leading to the outage.
• We designed our systems to gracefully handle performance degradation from dependent services. However, due to a misconfiguration, we were unable to prevent the outage.
• To restore ad serving and prevent cascading failures, we restarted the services by provisioning additional resources.
• We reproduced the failure in a test by degrading the availability of the internal service, proving the misconfiguration caused the cascading failures. We have since rolled out a fix to the configuration globally.
• We are conducting a complete review of all our processes and production configurations to prevent this from happening again.
It is unknown if Google will offer any compensation for those websites and publishers that were affected by the outage.
Latest posts by Jennifer Slegg (see all)
- Google Updates Quality Rater Guidelines: Reputation for News Sites; Video Content Updates; Quality for Information Sites - September 13, 2019
- Google Makes Major Changes to NoFollow, Adds Sponsored & UGC Tags - September 10, 2019
- Google Updates Quality Rater Guidelines Targeting E-A-T, Page Quality & Interstitials - May 17, 2019
- Google Local Service Ads Display Pricing Estimates for Specific Locations - August 31, 2018
- Google Testing “Relevant History” Section in Mobile Search Results - August 31, 2018