When Google stopped serving DoubleClick Ads on publisher websites last Wednesday, it turns out that more than 55,000 publishers were affected.  The sites,  which included major websites such as Forbes.com, The Wall Street Journal and The Guardian, caused issues beyond simple ads failing to load, as it seemed to cause some websites to hang as they tried to connect to the DoubleClick servers for ads.

The outage, which lasted from 5:45 am through 7:31am PST, was the first major outage for DoubleClick’s DoubleClick for Publishers (DFP) program.

Google’s Neal Mohan, VP, Display & Video Advertising and Scott Silver, VP, Engineering sent an email to publishers explaining the cascading issue that caused the loss of ad serving.

The details:
• The DFP ad server relies on an internal service that began degrading in performance. This caused a cascading failure on DFP ad servers, leading to the outage.
• We designed our systems to gracefully handle performance degradation from dependent services. However, due to a misconfiguration, we were unable to prevent the outage.
• To restore ad serving and prevent cascading failures, we restarted the services by provisioning additional resources.
• We reproduced the failure in a test by degrading the availability of the internal service, proving the misconfiguration caused the cascading failures. We have since rolled out a fix to the configuration globally.
• We are conducting a complete review of all our processes and production configurations to prevent this from happening again.

It is unknown if Google will offer any compensation for those websites and publishers that were affected by the outage.