The Cause Was Fixed. The Failure Continued.
Something bought on an ordinary day decides how long the worst day lasts. It appears on no dashboard.
On 18 November 2025, Cloudflare deployed a corrected configuration file across its network at 14:30. Its own published incident timeline records that all services were resolved at 17:06.
Two hours and thirty six minutes, with the cause already gone.
Every efficiency an institution buys is recorded on one side of the ledger. The load that same decision creates on the day the system is in trouble is recorded nowhere, because nobody has a unit for it.
The number that is not on the dashboard
A system has a capacity. That number is measured, reported and understood. It sits in the capacity plan and behind the sizing decision.
There is a second number.
In 2021, four engineers described a failure pattern they had observed across a decade of running systems at very large scale. Nathan Bronson, Abutalib Aghayev, Aleksey Charapko and Timothy Zhu called it metastable failure. Their model has three states.
In the first, load is low enough that the system absorbs a shock and returns to normal on its own. In the second, load is higher, the system is entirely healthy, and everything works. In the third, a shock has arrived and throughput has collapsed, and it stays collapsed after the shock has gone, because a feedback loop inside the system now sustains the overload by itself.
The middle state is where the argument lives. A system can sit there for years. It is not degraded. It is not struggling. Every metric reads normal, because everything is normal. The authors are explicit that many production systems are deliberately run in that state all the time, for the straightforward reason that it is far more efficient than the alternative.
Their worked example is a web application in front of a database. The database handles three hundred queries per second. Put a cache in front of it with a ninety per cent hit rate and the application serves three thousand. The advertised capacity is three thousand. The load below which the system would heal itself after losing that cache is three hundred.
A gap of ten times. One of those numbers is on the dashboard. The other only exists during a failure that has not happened yet.
That second number is obtainable. It is not observable. The distinction matters, because obtaining it means deliberately driving a system past the point where it can return on its own, and then watching whether it does. Almost nobody schedules that against an estate carrying customer money, which is why the number is usually absent rather than unknowable.
The efficiency and the exposure are the same purchase
The mechanism that sustains a failure is usually the thing installed to make the system better.
Retries mask transient faults, and multiply work at the moment there is no capacity to spare. Caches multiply throughput, and multiply the load arriving at the database when they empty. Detailed error handling makes an incident easier to diagnose, and consumes the resource that is already exhausted.
Cloudflare’s account of November is a clean example. Their bot management module preallocates memory for machine learning features, which is a performance optimisation. Preallocation requires a fixed ceiling. The ceiling was two hundred. Actual use was around sixty. A change to database permissions caused a query to return duplicate rows, the generated file went past two hundred, and the code path failed.
Nobody chose a two hundred feature limit. They chose preallocation. The limit arrived with it, unpriced, and sat there for years being harmless.
In the same incident, network latency rose because the debugging and observability systems consumed large amounts of processor time automatically enriching uncaught errors with additional detail. The tooling built to explain a failure was helping to sustain one.
The margin between recovering and not recovering is thinner than most capacity conversations assume. In a controlled reproduction published in 2022, the same researchers held the workload constant and varied only the shock. Ten seconds at seventy eight per cent loss of processor capacity produced a dip and a full recovery. Ten seconds at eighty per cent never recovered. Two percentage points decided it. Holding the shock at eighty per cent and shortening it by one second also produced recovery.
Four nights, many years ago
Many years ago I watched a batch estate come apart after a data migration.
The root cause was not subtle once it was found. The migration wrote a subset of records into the target in a state the downstream jobs had never been built to handle. Those jobs performed no validation at that boundary. They had never needed to, because for as long as anyone could remember the upstream had produced well-formed data, and that guarantee had never been written down. It was an assumption held in code, inherited by everyone who touched it afterwards.
So the run did not skip the bad records. It failed on them, and stopped.
That coupling had been in the estate for years. It had never mattered, because the conditions that would expose it had never arrived on an ordinary night. On paper the estate was within capacity, and on paper it always had been.
What stayed with me is not the first night. It is the second.
An overnight window is fixed. The work inside it is not. A run that does not complete leaves arrears, and the following night carries its own load and those arrears through the same window. The second night begins further behind than the first. The third begins further behind than the second.
By the second night the corrupted records had been found and corrected. The cause was gone. No new defect had been introduced. The estate kept failing because the thing driving it was no longer the bad data. It was the arithmetic of a fixed window carrying a growing debt, and the recovery runs were themselves part of the load. That is the sustaining effect, in a batch estate rather than a distributed service, with the same shape. It took several cycles, and a deliberate decision to cut back what the estate was asked to do each night, before the arrears cleared.
The migration decided when the failure started. It did not decide how long it lasted. That was decided by how much slack sat in a nightly window, settled years earlier by people making a reasonable trade between cost and headroom on days when nothing was happening.
None of them were wrong. None of them were asked the question.
The fix that became the amplifier
Resilience thinking assumes that remediation makes things better. It is not reliable.
Spotify had an incident sustained by retries. Afterwards, engineers added substantial logging to the error path, so that next time they would understand what had happened. In the following incident, that logging raised the cost of every retry, which added load, which caused more retries. The fix from the last failure was the amplifier in the next one.
The team had repaired their ability to explain the failure. The loop that produced it was untouched, and the explanation machinery was now inside it.
An engineering reader will already be reaching for the mechanism that exists for exactly this trade. Site reliability engineering uses an error budget: two teams jointly set, from a service objective, how unreliable a service is permitted to be over a quarter. It removes the politics from a negotiation previously settled by whoever argued best, and it is the most serious attempt the industry has made to put a number on this.
My reading is that it does not close the gap, and the reason is the denomination. An error budget is stated in unavailability already observed. It records how much failure has occurred. It does not say how far the system would fall, or whether it would return unaided. A service can sit comfortably inside its budget and one shock away from a state it cannot leave.
Worse, a healthy budget is evidence of a quiet period, and a quiet period is when the amplification is least visible. The budget then reads as permission to spend more of a margin nobody has measured. That is my extension, not Google’s. The same body of practice answers the problem a few chapters later, which is where this ends up.
The circular dependency
The institutional version of this argument is the one that reaches a board, and Cloudflare published it about themselves.
After two global outages inside six weeks, they declared a company-wide programme ranked above all other work. One of its three workstreams concerned neither configuration nor failure modes. It was the removal of circular dependencies.
Their account is candid. During both incidents their own security systems prevented team members from reaching the tools they needed. They are a security company, every tool sits behind authentication and fine-grained access control, and they say directly that this is right. It also slowed them down when speed mattered most.
The consequence reached customers. Turnstile, their bot verification product, sits on the login flow for the Cloudflare dashboard. When Turnstile became unavailable, customers without an active session could not log in to make changes at the moment they most needed to.
The control protecting the front door was served by the network that was down.
Nothing about that shape is specific to a technology company. A recovery path runs through incident tooling, access management, an approval workflow, a communications channel, and a small number of people who understand a system well enough to make a judgement at two in the morning. Every one of those dependencies is a sensible efficiency on an ordinary day, and invisible in a design review, because on an ordinary day it works.
The question I have taken from this is narrow, and I now ask it of any test I am shown. Did the system recover, or was it recovered? If the shock was removed by hand and the system was helped back, the test measured the shock. It did not measure the loop.
In fairness, and because a case that only flatters is not evidence: Cloudflare suffered a further outage on 20 February 2026 affecting bring-your-own-IP customers, while that programme was still running. Their assessment that the completed work would have prevented the two 2025 incidents is their own assessment of themselves.
You cannot measure it. You can bound it.
The 2022 paper states the relationship formally. The load a system can safely carry equals its normal capacity divided by the maximum amount by which work is amplified when the system is in trouble. Two terms. One is measured continuously. The other is not observable during normal operation.
It does not have to be.
The amplification factor is only an unknown if it is left to emerge. Capped by design, it stops being a discovery and becomes a decision.
The same paper puts it in one line: a retry policy with no cap leaves the system with effectively no stable region at all.
Cap the amplification, and write the cap down as a number. Google’s published practice uses two caps together. A per-request budget of three attempts, after which the failure passes to the caller. And a per-client budget, where a client retries only while retries run below ten per cent of its total requests. The first alone allows volume to grow to just under three times. The two together bring it to roughly 1.1 times. That is an amplification factor named and bounded rather than inherited. It assumes the operation is safe to repeat, which is its own design commitment.
Retry at one layer only. A rejected request is retried by the layer immediately above the one rejecting it, and by nothing higher. When a layer decides a request cannot be served, it returns an explicit instruction not to retry rather than an ordinary error. Without that rule each layer multiplies the one below it and the growth becomes combinatorial. One line of architectural standard, and it is checkable.
Decide in advance what gets shed. Google classifies every request into one of four criticality levels, from the most user-visible down to work where occasional full unavailability is expected. The classification is set as close to the customer as possible, propagates automatically down the call stack, and drives what is rejected first under pressure. The institutional translation is plain. What an institution stops doing on its worst day is either decided in advance, in daylight, by people with the authority to decide it, or it is settled at three in the morning by whoever is awake. Both are decisions. Only one is governance.
Default to the last good state rather than to nothing. Cloudflare’s answer to their own failure is that a system unable to read a new configuration should refuse it and keep the previous one, and where that is impossible, that each case is decided in advance on whether reduced functionality beats none. Had that existed in November, an unreadable file would have cost them real-time tuning of a detection model. It would not have cost them the network.
Take the recovery path out of the failure domain. They audited the tools needed for visibility, debugging and production change, and built backup authorisation pathways for eighteen services. Then they ran an engineering-wide drill on 7 April 2026 with more than two hundred people, on the reasoning that automation keeps a pathway working, but only practice builds the reflex to use it under pressure.
Making it survive the people who learned it
Every one of those is a rule somebody had to be taught by an outage.
A lesson living in the judgement of the engineers who were in the room has a decay curve. They move teams. They leave. The rule survives as folklore, then as a habit nobody can explain, then not at all. Three years later the same shape appears in a new service and nobody present recognises it.
Cloudflare took those lessons out of people’s heads and wrote them into a single internal standards repository, produced through a formal proposal process, in a fixed format: if you need this, do that, with the reasoning linked. It is mandatory across the entire codebase, and enforced automatically at code review, so a change that violates a rule is stopped there rather than found later.
One of their rules requires a service to validate that its upstream dependencies are in an expected state before it processes anything.
That is precisely the rule the batch estate did not have. It had been missing for years. No individual was ever going to be held to account for its absence, because no individual had ever decided it.
I wrote recently that the most durable work an institution does leaves no trace, and that this is exactly why it goes uncredited. The codified rule is how that work acquires a trace. The judgement stays invisible. The rule it produced does not. It has an author, a reason attached, and an enforcement point that does not get tired, does not leave, and was not in the room when the lesson was learned.
That is the difference between an institution that knows something and an institution that will still know it in a decade.
What this costs
An institution that cannot state its amplification factor is not running without risk. It is running with an unpriced liability, and the price is paid in duration.
Duration is what customers experience. Duration is what a board is asked about afterwards. And duration is not set by the size of the event. It is set by decisions taken years earlier, in ordinary conditions, by capable people optimising something sensible, none of whom were asked what the decision would cost on a day when the system was already in trouble.
Three questions are cheap to ask and uncomfortable to answer. What is the cap on how much extra work this system creates when it is already struggling, and is that cap written down anywhere. What does this system stop doing first, and who decided. And when we recovered last time, did the system come back, or did we carry it back.
None of them need a new platform. They need somebody to ask while nothing is happening, which is the only time the answers can still be changed.
Nothing is decided on the worst day. The worst day only collects.
If you found this useful, the likelihood is someone you know is asking the same question. Pass it on.
Sources
Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu. “Metastable Failures in Distributed Systems.” HotOS ‘21, ACM. https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf
Lexiang Huang, Matthew Magnusson, Abishek Bangalore Muralikrishna, Salman Estyak, Rebecca Isaacs, Abutalib Aghayev, Timothy Zhu, Aleksey Charapko. “Metastable Failures in the Wild.” OSDI ‘22, USENIX. https://www.usenix.org/conference/osdi22/presentation/huang-lexiang
Cloudflare. “Cloudflare outage on November 18, 2025.” https://blog.cloudflare.com/18-november-2025-outage/
Cloudflare. “Code Orange: Fail Small.” 19 December 2025. https://blog.cloudflare.com/fail-small-resilience-plan/
Cloudflare. “Code Orange: Fail Small is complete.” 1 May 2026. https://blog.cloudflare.com/code-orange-fail-small-complete/
Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy, eds. Site Reliability Engineering. O’Reilly, 2016. Chapter 3, “Embracing Risk,” and chapter 21, “Handling Overload.” https://sre.google/sre-book/embracing-risk/https://sre.google/sre-book/handling-overload/