Finding the Single Point of Failure Before It Finds You

Finding the Single Point of Failure Before It Finds You

The server was down 40 minutes. The business was down three days. The difference: the router that failed was the only one, and nobody had ever doubled it up.

Nearly every serious outage I've reviewed traces back to the same shape. It's not the server that's the real story — it's one component with no backup, and everything downstream going dark in sequence. Finding it before it fails isn't a budget problem. It's a mapping problem, and most businesses have never done the mapping.

What a single point of failure actually is

A single point of failure (SPOF) is "a part of a system that would stop the entire system from working if it were to fail" — implying no redundant backup behind it. Wikipedia's definition happens to be, unusually, the exact one engineers use.

Notice what it leaves out: how likely the component is to fail. Low odds just mean you'll be unlucky less often. A 0.01% annual failure rate is still a SPOF if there's nothing behind it.

Engineers name three recurring shapes. A bottleneck throttles rather than stops — one payment gateway, one person reviewing every deployment. A cascading failure is one failure tripping the next dependent component until it becomes a full outage. An Achilles' heel is the SPOF nobody suspected, invisible until it's the only thing that mattered.

The six places SPOFs hide in an SME

Most businesses have all six. Most owners can name maybe two.

One server or database with no failover. A nightly backup nobody has restored, no warm replica. A dump you've never restored is a hypothesis, not a backup — see verifying Postgres backups actually work.

One network link or router. One ISP line, one router feeding the office. A crew cuts fiber three streets over and the operation goes offline no matter how good the servers behind it are.

One vendor or SaaS dependency. Payment processing, transactional email, the CRM the sales process runs on. A vendor outage or API change with no fallback turns their incident into yours.

One person who knows everything. The bus-factor problem: one engineer holds the deploy credentials or the only mental model of the architecture. A bus factor of one is a critical SPOF by definition; three to four is healthy.

One process approver. One signer for purchase orders or refunds, no delegate. When they travel, the workflow queues behind them — a human bottleneck shaped like a database with no replica.

One power source. One utility feed, an untested UPS, a generator nobody starts on schedule. Power sits underneath every other SPOF here: replicas don't help if they all lose power at once.

Four outages, one pattern

In April 2011, a routine capacity upgrade in AWS's US-East region misrouted traffic onto the lower-capacity redundant EBS network instead of the intended router, cascading into a multi-day outage — AWS's own post-mortem traces exactly how.

In October 2021, Meta's DNS servers were configured to withdraw their own route advertisements whenever they couldn't reach Meta's data centers. A maintenance command severed backbone connectivity, and DNS pulled itself offline too, taking Facebook, Instagram, and WhatsApp down for roughly six hours. The safety mechanism itself was a self-referential dependency with no way to signal "unhealthy" without also going dark.

In July 2024, CrowdStrike shipped a channel file whose structure didn't match its sensor code's expectations, an out-of-bounds read that crashed roughly 8.5 million Windows machines worldwide, by Microsoft's own estimate. CrowdStrike's root-cause report shows one file, pushed to every customer at once, becoming the SPOF for an entire software category.

And in May 2017, British Airways lost check-in and ground systems for three days after an electrical engineer disconnected the data center's uninterruptible power supply. As widely reported, thousands of passengers were affected, with then-chairman Willie Walsh pinning the outage on that one disconnection.

A misrouted network, a self-inflicted DNS design, a bad software push, a power surge — four failure types, one pattern. Every organization had redundancy on paper, and each had a single dependency all of it ran through unexamined.

Map your own SPOFs with a dependency walk

Pick two or three flows the business can't survive losing — order intake, payment, delivery confirmation. Trace every dependency each one touches: services, database, queue, DNS, network path, third-party APIs.

Anything appearing once with nothing behind it is a candidate SPOF, operational dependencies included. A single deployment pipeline, or the one person who can run it, is a SPOF in the same sense as an un-replicated database.

That walk usually surfaces five to ten SPOFs in an afternoon, most never written down.

Cheap mitigations vs. full high availability

Not every SPOF needs the same fix, roughly in order of cost:

• Free or near-free: a documented backup approver, written runbooks, a second person with locked-down access to production credentials.
• Cheap, one-time: a second ISP line or LTE failover router, a monthly UPS/generator load test, a status page on a different provider.
• Real spend: a warm database replica with restore testing, a fallback payment processor actually tested, redundant links from two providers.
• Full high availability: multi-region active-active, automatic failover, 24/7 on-call — expensive and heavy, right for some, wrong for most.

When redundancy is not worth the cost

Google's own site reliability practice argues 100% reliability is "probably never the right reliability target," because cost doesn't scale linearly with reliability — "an incremental improvement in reliability may cost 100x more than the previous increment."

The skill isn't eliminating every SPOF. It's matching the mitigation tier to what an hour of downtime actually costs.

For a firm with real revenue riding on every transaction, that can easily run into the hundreds of thousands of dollars. That's cheap insurance for a fallback processor and a tested replica.

For a ten-person shop, the same hour might cost a few hundred dollars in lost sales, and a multi-region build is money better spent on the product itself. Size the mitigation to what your own outage would cost, not to what happened to a company a hundred times your size.

A self-audit for this week

1. Pick the two or three flows the business cannot survive losing.
2. Walk every dependency; flag anything appearing once with no backup.
3. Write down what each SPOF would cost per hour if it failed today.
4. Fix the free ones this week: backup approver, runbook, second credential.
5. Price the expensive ones against that hourly cost before you commit.

Most of that list costs nothing but an afternoon. The businesses that get hurt aren't the ones that skipped multi-region failover — they're the ones that never did the walk at all. If you want a second set of eyes on where your own map has gaps, get in touch.