Everything Fails. Design for Containment.
Why failure is always funded after the fact and never before it.
How much of your engineering investment last year went to containing failure versus preventing it.
Most programmes cannot answer that question precisely because they have never framed it that way. And what is not named in a budget is not a priority. It is a risk.
Containment design is the discipline of bounding failure so that it cannot propagate beyond the capability where it originates. The unit of that discipline is the fault domain. A fault domain is an explicit boundary drawn around a capability or component. Within that boundary isolation governs what stays inside when something goes wrong. Failure within a well-isolated fault domain cannot reach the capabilities beside it. A circuit breaker enforces that isolation operationally. When a component begins behaving outside its designed parameters the circuit breaker stops propagation before the blast radius expands. The blast radius is the scope of customer and business impact when a failure occurs. Containment design does not prevent failure. It determines how much of the system and how many customers are affected when it arrives.
A card payment authorisation system with well-designed containment will degrade to stand-in processing, a reduced but functional alternative, when the primary authorisation engine fails. The customer continues to transact. The failure is isolated. It does not cascade into account management, statement generation, or any adjacent capability.
A system without containment design will execute four million erroneous orders in 45 minutes when a single misconfigured server activates dormant code. There is no circuit breaker. The blast radius is the entire operation.
The difference between those two outcomes is not the quality of the engineering. It is the presence or absence of a deliberate containment design.
Most organisations understand this in principle. Almost none fund it in practice. The reason is cultural before it is financial.
The Prevention Illusion.
When something fails, two things happen simultaneously. The technical response begins. And the cultural response begins.
The technical response is about what failed. The cultural response is almost always about who failed. And in organisations where the cultural response dominates, the technical response is shaped by it. The postmortem becomes a performance. The action items become a record of accountability rather than a plan for design improvement. The fault domain gap that would require investment to address is noted and deferred.
This is not cynicism. It is organisational physics. In a blame culture, naming anticipated failure is indistinguishable from accepting personal responsibility for it. No engineer or architect will voluntarily design a fault domain that explicitly anticipates the failure of the capability they own. The political cost is too high. The immediate reward is invisible.
The result is a programme optimised for prevention. Zero downtime targets. Ambitious reliability standards. All built on the assumption that careful enough design can avoid failure. The assumption is wrong in the general case. And it produces a specific consequence. Systems that have not been designed for the failures they will inevitably experience.
Earlier in this series we established that commitment is irreversible and architectural consequence is permanent. If both are true then the question for any system is not whether it will fail. It is what the consequence of that failure will be when it does. Containment is the engineering discipline that governs that consequence. The funding gap is the leadership failure that prevents it from being built.
The Funding Gap.
Two patterns appear consistently across organisations. Together they explain why the gap between knowing failure is inevitable and designing for it persists.
Pattern one. The postmortem that produced a document but not a design change.
An incident occurs. A blameless postmortem is held. The root cause is identified. Action items are written. The urgency dissipates. The next sprint is committed to feature delivery. The containment improvement sits in the backlog. It is never prioritised. The same fault domain fails again under different conditions six months later.
This is a framing failure before it is a prioritisation failure. The postmortem is framed as a learning exercise rather than a design review. Learning is cultural. Design is funded. What is not funded is not built.
Pattern two. The incident review that identified the fault domain gap but never funded the containment work.
The gap reaches a governance conversation. The fault domain risk is named. A containment investment recommendation is made. The recommendation is declined. The business case for feature delivery is more visible than the business case for containment. Customer outcomes are legible. Blast radius reduction is not.
This is where the framing failure becomes a leadership failure.
The containment work was framed as competing with customer delivery. That framing is precisely backwards. Every failure that was not contained ultimately destroyed the customer outcomes the feature delivery was building toward. The customer who cannot complete a card payment because a fault domain failure propagated across an uncontained boundary has not received a customer outcome. Containment design is customer outcome delivery. It is simply the part of customer outcome delivery that is invisible until it fails.
This is what that invisibility looks like from the inside.
The Boundary That Was Not Designed For.
As Engineering Lead for Card Payment Authorisation and Clearing, following a Visa compliance release, I encountered the containment design gap in its most instructive form.
Every internal measure showed the system as healthy. Metrics nominal. Alerts silent. Scheme partners were experiencing timeouts.
The fault domain failure was at the boundary. The observability had been designed to measure health from inside the domain. It had not been designed to measure the blast radius of a failure as experienced from outside it. The system appeared contained because we were measuring from the wrong position.
The containment question the incident revealed was this. What is the failure mode at this service boundary and has it been explicitly anticipated. The answer was no. The failure mode had not been designed for. The blast radius was discovered rather than bounded.
This is the pattern I have observed consistently across programmes. The fault domain is technically defined. The failure mode at the boundary of that fault domain is not. The containment design exists within the domain. The boundary between domains is where the most consequential failures find their home.
What Uncontained Failure Actually Costs.
On August 1 2012, Knight Capital Group, then responsible for approximately 10% of all trading in US equity securities, experienced what the absence of containment design produces at scale.
A deployment error left one of eight trading servers running deprecated code. When the market opened, orders triggered the dormant code. No circuit breaker. No isolation infrastructure to bound the blast radius of a single misconfigured server. In 45 minutes the system executed 4 million orders across 154 stocks. Ninety-seven automated error emails were generated and went unaddressed. The loss was 440 million dollars. A company built over 17 years effectively ceased to exist before lunch.
The failure was not the deployment error. Human errors in deployment are foreseeable. The failure was the absence of containment design that would have bounded what a single misconfigured server could do before the blast radius became irreversible.
The SEC enforcement action that followed named the most important detail for this argument. The firm reacted to prior events too narrowly and did not adequately consider the root causes of previous incidents. Knight Capital had experienced prior incidents. They produced postmortems. They responded. The responses were too narrow. The root cause was not funded. The fault domain that destroyed the firm was the predictable consequence of a pattern of learning from failure rhetorically without designing for it structurally.
Knight Capital did not fail to fund failure. It funded the remediation after each prior incident. It simply never funded the containment before the incident that ended it.
What Designing for Failure Looks Like.
If we assume this will fail, what must we build before it does.
That question governs the organisations that design for failure rather than against it.
Netflix built Chaos Engineering as an institutional response. Their approach of deliberately injecting failure into production systems rests on one insight. If you do not test your containment design you do not know whether it works. And if you do not know whether it works your reliability targets are aspirations rather than guarantees.
Amazon’s cell-based architecture applies the same philosophy structurally. The system is divided into isolated cells such that a failure within one cell cannot propagate to adjacent cells. The blast radius of any failure is bounded by design before deployment rather than discovered during an incident.
Both approaches rest on the same foundation. Failure is not a risk to be managed. It is a certainty to be contained. That acceptance requires an investment decision before an incident makes it urgent.
The following questions make that investment decision practical rather than philosophical.
The Containment Design Framework.
Five questions. Designed to work at two levels simultaneously.
For engineers and architects.
What are the fault domains in this architecture. If they are not explicitly designed their blast radius is undefined.
What is the failure mode at each service boundary. If it has not been anticipated it will be discovered during an incident. The boundary between domains is where the most consequential failures find their home.
What is the circuit breaker. If a component begins behaving outside its designed parameters what is the mechanism that stops propagation before human intervention is possible. Knight Capital’s 97 unaddressed error emails are the illustration of its absence.
For boards and CTOs.
What is the blast radius of the organisation’s highest risk failure scenario calculated from a customer harm perspective. Has the board seen that number. If not the board has accepted an exposure it has not priced.
If the regulator reviewed the organisation’s response to prior incidents tomorrow would they find root causes addressed or responses too narrow. The SEC found the latter at Knight Capital. The question is whether they would find the former at yours.
The Leadership Decision.
Containment design is not an engineering decision. It is a leadership decision about what kind of relationship the organisation has with failure.
The leader who frames all investment as customer outcome delivery and containment work as engineering overhead has not made a customer-centric decision. They have accepted a framing that will eventually destroy the customer outcomes they are protecting by not funding it.
The customer outcomes are visible. The blast radius of their absence is not. Until it is.
The containment work that has not been funded is a capital allocation decision that has already been made. The postmortem that did not produce a design change is a governance decision that has already been made. The blast radius that has not been calculated is an exposure that has already been accepted.
Failure is coming. The blast radius is a choice. And the funding decision is a leadership one.
The Sutra
Failure is always funded. The question is whether you fund the containment before it or the remediation after it.