"I'm panicking, what happened?" Cloudflare outage causes global internet chaos

The incident once again highlights the global internet's heavy reliance on a handful of infrastructure providers.

Written by: Zhao Yuhe

Source: Wallstreetcn

On Tuesday morning Eastern Time, internet infrastructure service provider Cloudflare reported that its global network was experiencing anomalies, resulting in access failures such as "internal server error" on a large number of websites, including the social media platform X. Users were unable to access many websites and services, including retail, e-commerce, social media, financial services, and transportation-related platforms. The company later claimed that the issue was resolved in less than four hours.

During the outage, some functions of X were interrupted, and many websites also experienced access issues. According to data from the outage tracking platform Downdetector, in addition to X, a large number of sites were affected, with the number of related reports continuing to rise. Users visiting sites such as X, ChatGPT, DoorDash, IKEA, and the New York City Metropolitan Transportation Authority (MTA) all saw error messages related to Cloudflare.

Subsequently, a female spokesperson for Cloudflare stated that around 6:20 a.m. Eastern Time, one of their services experienced an abnormal surge in traffic, causing errors in the traffic passing through the company's network.

Another Cloudflare spokesperson, Jackie Dutton, said in a statement that the issue was caused by an automatically generated configuration file used to manage threat traffic, and the fix took less than four hours. The company stated that core remediation measures had been deployed, but cautiously noted that the system "still needs time to fully stabilize."

Dutton stated:

"The number of entries in this file exceeded the expected size, triggering a crash in the software system responsible for handling traffic for some Cloudflare services."

The statement said there was no evidence that the incident was related to a cyberattack or malicious activity.

The scope of the outage was extremely broad. Downdetector reported on its platform that during the Cloudflare outage, "the total number of reports for affected services exceeded 2.1 million," indicating that this incident has become one of the more severe infrastructure-level outages in recent years.

After the incident, Cloudflare's stock price plunged as much as 7% at Tuesday's opening, before narrowing its losses.

The digital asset industry also responded. Binance co-founder and former CEO Zhao Changpeng posted on X: "Blockchain kept working," implying that decentralized systems were unaffected by the incident.

By 12:15 p.m. Eastern Time, Cloudflare stated that the system was gradually recovering, but some regions around the world might still experience access errors, performance degradation, or login issues. The company will continue to update the repair progress on its status page.

Overreliance on a Few Companies

In recent years, there have been multiple incidents where problems with digital infrastructure providers have crippled global internet usage. Amazon Web Services (AWS), CrowdStrike Holdings Inc., and Microsoft have all experienced similar incidents, further highlighting the global internet's significant reliance on services provided by a handful of companies.

For ordinary users, services from Cloudflare and AWS are almost "invisible," but their tools support a vast number of websites and services that consumers use every day.

Last month, an AWS outage paralyzed parts of the internet, rendering websites and apps for millions of users unusable, disrupting retail sales, social media, and financial services, and affecting many businesses. Last year, a vulnerability in a tool used by cybersecurity company CrowdStrike caused a massive crash of computer systems worldwide, leading to thousands of flight delays and cancellations, and throwing government agencies and large enterprises into chaos.

Graeme Stewart, an expert at California cybersecurity company Check Point Software, said such incidents highlight the internet's overreliance on a few infrastructure providers.

He said:

"Many organizations still rely on the same path for all critical services, and there is no truly effective backup. Once this path has a problem, there is no fallback plan. This is the issue we keep seeing."

Alan Woodward, a cybersecurity professor at the University of Surrey, said Tuesday's outage once again demonstrated the internet's heavy reliance on "a few players." He described Cloudflare as "the biggest company you've never heard of."

"People have no choice but to rely on these few big companies."

Chief Technology Officer Apologizes

Cloudflare Chief Technology Officer Dane Knecht apologized for the incident. He wrote on X:

"When the Cloudflare network has issues that impact the massive traffic relying on us, we let down our customers and the entire internet. The issue itself, the impact, and the time it took to resolve are all unacceptable. We've already started working to ensure this doesn't happen again, but I know today has indeed caused trouble for everyone. Our customers' trust is most important, and we will do everything we can to win it back."

Cloudflare has experienced similar outages several times in recent years.

In July 2019, a bug in Cloudflare's software caused some network modules to excessively consume computing resources, taking offline thousands of websites that relied on Cloudflare (including Discord, Shopify, SoundCloud, and Coinbase) for up to 30 minutes. In June 2022, Cloudflare experienced an outage affecting traffic at 19 of its data centers, causing multiple major websites and services to go down for about an hour and a half.

Cloudflare's software is used by hundreds of thousands of companies worldwide as a buffer layer between enterprise websites and end users, protecting websites from traffic attacks or outages caused by traffic surges.

Last year, a faulty software update released by cybersecurity company CrowdStrike caused millions of devices running Microsoft Windows to crash, creating widespread chaos in industries such as aviation, banking, and healthcare.

This CrowdStrike outage stemmed from a bug in its product that runs at the lowest level of customers' computers. Cloudflare, on the other hand, protects internet infrastructure such as websites and platforms, so when Cloudflare goes down, many popular websites become directly inaccessible or experience anomalies. Cloudflare is mainly responsible for "keeping websites online and fast," while CrowdStrike focuses on protecting computers and servers from attacks.