RAID Is Not a Backup: What Array Failure Actually Costs

RAID Is Not a Backup: What Array Failure Actually Costs

A four-bay NAS in an office cupboard holds ten years of project files. It ran RAID 5 the whole time, and the people relying on it had been told that meant the data was safe. Two disks reported errors in the same week, the volume went offline, and the recovery quote arrived with a number nobody had budgeted for.

Nothing behaved outside its design. RAID 5 tolerates one disk failure. The second one was always out of scope. The gap was never in the hardware — it was in what everyone believed the hardware had promised.

RAID buys a window, not a copy

Redundancy across disks answers exactly one question: can the array keep serving reads and writes after a drive dies? That is uptime. It buys a window in which to replace the failed disk without stopping work. It is the spare tyre, and a spare tyre does nothing about a stolen car.

Everything else on the list of things that destroy data walks straight through it:

• A deleted file is deleted on every member disk at once.
• Ransomware encrypts through the array like any other write.
• A dropped table, a bad migration, an application writing garbage — all faithfully replicated.
• Filesystem corruption sits above the RAID layer and is copied below it.
• A surge, a fire, or a theft takes the whole chassis.
• The controller itself can fail and take the array with it.

A backup is a copy that exists at a separate point in time, on separate hardware, that a single bad event cannot reach. RAID is none of those three. Same moment, same box, same blast radius.

The rebuild is where arrays actually die

Every drive datasheet carries an unrecoverable read error rate. Consumer SATA disks are typically specified at fewer than one URE per 10^14 bits read; nearline SAS often claims 10^15. That first figure is about 12.5 TB of reads per expected error.

An eight-disk RAID 5 built from 4 TB drives has to read 28 TB of surviving data to rebuild one member. That is 2.24 × 10^14 bits — call it two expected read errors at the consumer spec, during the one operation that has no redundancy left to absorb them.

The spec number is a ceiling the vendor commits to, not a measurement, and observed field rates are generally better than the datasheet. Treating a rebuild as doomed is wrong. Treating it as routine is also wrong.

What the error does depends on your stack. A hardware controller may drop the drive out of the array and abort the rebuild, turning one dead disk into a dead volume. Linux md is more forgiving: it logs the bad sector, continues, and you lose whatever file lived in that block. Which behaviour you have is a thing to know before the day it matters.

Then there is time. A 16 TB drive resilvering at a real-world 150 MB/s needs about thirty hours if nothing else touches the array. Production I/O and rebuild throttles routinely stretch that to two or three days. For every hour of it the array has zero redundancy, and every surviving disk is being read end to end at full load.

Disks bought together fail together. Large-scale field studies of production populations have found replacement rates several times the datasheet MTBF, and failures clustered in time rather than spread evenly across a fleet. The neighbour of a dead drive is the most likely next candidate, and the rebuild hands it the heaviest week of its life.

Parity knows something is wrong, not what

Scrub a RAID 5 array and it reads every stripe, recomputes parity, and counts mismatches. That count says the stripe is inconsistent. It cannot say whether the data block is wrong or the parity block is.

Linux md resolves the ambiguity by assumption. Writing repair to /sys/block/md0/md/sync_action recomputes parity from the data blocks and writes it back, which is right when parity drifted and quietly launders the evidence when the data drifted. RAID 1 repair copies the first drive over the others. Neither is a bug — there is no information available to do better.

The write hole is the same shortage of information in a different place. A stripe update touches a data block and a parity block. Power loss between the two leaves the stripe internally inconsistent, and nothing records that fact. Rebuild off that parity months later and you get reconstructed blocks that were never real. mdadm closes this with a write journal or a partial parity log; hardware controllers close it with a battery-backed cache, which works until the battery ages out unnoticed.

Checksummed filesystems remove the ambiguity by storing a hash with every block. ZFS and btrfs read a block, check the hash, and when it fails they know which mirror or which reconstruction to trust. That is the line between detecting inconsistency and correcting corruption, and it is the strongest argument for ZFS over a hardware card for data you intend to keep.

Whatever you run, schedule the scrub monthly and alert on the counter:

cat /sys/block/md0/md/mismatch_cnt

A nonzero result on a RAID 5 array is information you want on a quiet Tuesday, not halfway through a rebuild.

The controller is part of your on-disk format

Hardware RAID cards write their own metadata to the member disks: stripe geometry, member order, array state. The format is vendor-specific and mostly undocumented. When the card dies, the disks are simultaneously intact and unreadable, and recovery starts with sourcing the same card family on compatible firmware before anything can be mounted.

Software RAID does not have that failure mode. The Linux md superblock is documented and any Linux host will assemble the array. A ZFS pool imports on any machine running ZFS, including a different operating system. For a small business with no spare controller on a shelf, that portability outweighs the throughput the card was bought for.

Whichever you run, save the geometry now, while everything works:

mdadm --detail --scan
for d in /dev/sd[b-i]; do mdadm --examine "$d"; done
lsblk -o NAME,SERIAL,SIZE,MODEL

Keep the output, the controller model and firmware version, and a photo of which serial number sits in which slot. Store it off the array. Rebuilding those parameters after a failure is one of the most expensive parts of a RAID recovery, and today it costs you five minutes.

Why the recovery quote is a multiple, not a markup

A single failed disk is one job. Image it, then work on the image — the rule zero of any recovery.

An array is that job N times before anyone opens a file. Every member gets imaged, because reading a degraded array live risks the disks you still have. Then the geometry has to be established from the images: chunk size, member order, parity rotation (left-symmetric is the Linux default, controllers vary), data offset, and which member had already gone stale before the array stopped.

Only then can stripes be reassembled, and the output is only as complete as the inputs. RAID 5 missing one member reconstructs from parity, and every unreadable sector on the survivors becomes a hole in a file. RAID 5 missing two members, or a RAID 0 stripe missing any member, is arithmetic without a solution.

The most expensive failures are not the disks. They are the decisions made in the first hour: rebuilding onto the wrong slot, accepting a controller prompt to initialize the array, running a filesystem repair against a degraded volume. Each one writes fresh data over the layout the recovery depends on. The correct first move on a failed array is to power it down and photograph the slot order.

What to run instead

Pick the RAID level for the rebuild, not the capacity. Above roughly 4 TB per drive, RAID 6 or RAID 10 keeps redundancy during the rebuild window that RAID 5 spends naked. ZFS mirrored vdevs resilver only allocated data, which finishes sooner and reads the survivors less.

Mix your batches and keep a hot spare. Different purchase dates or different vendors break the correlation. A hot spare starts the rebuild at 3am instead of when someone reads the email.

Monitor the array, not the server. SMART attributes, mdadm --monitor or zpool status, and an alert when the array goes degraded. A degraded array nobody noticed is the standard prologue to a total loss, and it belongs in the same minimal monitoring setup as your uptime checks.

Run 3-2-1. Three copies, on two media types, with one offsite. The rule comes from photographer Peter Krogh's digital asset management work and has outlived every storage generation since. The current variant adds one copy offline or immutable, and zero errors on verification.

Verify the restore, not the job status. A backup that has never been restored is an untested assumption with a green tick next to it. Restore on a schedule, to a separate machine, and check the contents — the same discipline that makes a pg_dump into an actual backup.

Write the array-failure runbook before you need it. Power down, do not initialize, photograph the slots, image every member. Three lines on the wall of the server room have saved more data than any controller feature.

The short version

RAID answers "can we keep working while a disk is dead". Backup answers "can we get the data back". They are different questions, and buying an answer to the first has never been an answer to the second.

Size the array for its rebuild, put checksums under it, scrub it on a schedule, and keep a copy somewhere the fire cannot reach. Then test the restore, because the only backup that counts is one you have already put back.