This week's Internet blackout was caused by one user updating his settings

General
This week's Internet blackout was caused by one user updating his settings

Fastly, one of the Internet's largest content delivery networks (CDNs), went down this Tuesday, leaving millions of users unable to access certain websites, while 85% of the networks Fastly covers returned errors Tuesday morning, This was all due to a single user configuration update that discovered a bug that had been lying dormant in Fastly's infrastructure since mid-May.

In an interview with The Guardian, Nick Rockwell, Fastly's head of infrastructure and engineering, explained what actually happened to bring down the service and also apologized for the disruption. It seems unusual for such a large company to be so transparent about something like this, but it is certainly a welcome development.

Content delivery networks operate on the principle that the Internet is faster and more stable when users are physically closer. This results in faster downloads, improved security, and many other features.

It also means that there is a point of failure in case something goes wrong, and that is exactly what happened on June 8. [On May 12, we began rolling out software that introduced a bug that could be caused by certain customer settings under certain circumstances," Rockwell told The Guardian. Early on June 8, the customer pushed a valid configuration change that included a specific situation that triggered the bug."

Essentially, the bug was introduced into the system on May 12, but lay dormant until the customer updated the configuration on June 8; Fastly discovered the problem within a minute, and "within 49 minutes, 95% of the network was up and running normally."

There were, of course, financial consequences to users not being able to access the site, with SEO agency Reboot estimating that the downtime cost Amazon $32 million in lost sales. Gasp.

Categories