RAID Is Not a Backup: Risks and Recovery Costs

RAID combines disks so storage can keep working through some disk failures, depending on the RAID level. It cannot recover deleted files or undo ransomware. You need separate backups and tested restores for that. Also check disk-replacement procedures: rebuilding data creates a vulnerable period when disk redundancy is reduced.

RAID 5 tolerates one disk failure. A second simultaneous failure is outside that protection. The gap was never in the hardware — it was in what everyone believed the hardware had promised.

RAID buys a window, not a copy

Redundancy across disks answers exactly one question: can the array keep serving reads and writes after a drive dies? That is uptime. It buys a window in which to replace the failed disk without stopping work. It is the spare tyre, and a spare tyre does nothing about a stolen car.

Everything else on the list of things that destroy data walks straight through it:

• A deleted file is deleted on every member disk at once.
• Ransomware encrypts through the array like any other write.
• A dropped table, a bad migration, an application writing garbage — all faithfully replicated.
• Filesystem corruption sits above the RAID layer and is copied below it.
• A surge, a fire, or a theft takes the whole chassis.
• The controller itself can fail and take the array with it.

A backup is a copy that exists at a separate point in time, on separate hardware, that a single bad event cannot reach. RAID is none of those three. Same moment, same box, same blast radius.

The rebuild is where arrays actually die

Every drive datasheet carries an unrecoverable read error rate. Consumer SATA disks are typically specified at fewer than one URE per 10^14 bits read; nearline SAS often claims 10^15. That first figure is about 12.5 TB of reads at that stated rate. The specification is an upper bound, not a measured average.

An eight-disk RAID 5 built from 4 TB drives has to read 28 TB of surviving data to rebuild one member. That is 2.24 × 10^14 bits. The volume exceeds the 10^14-bit reference, but the specification alone does not predict two errors or prove a rebuild will fail. RAID 5 has no remaining redundancy during this rebuild.

The spec number is a ceiling the vendor commits to, not a measurement, and observed field rates are generally better than the datasheet. Treating a rebuild as doomed is wrong. Treating it as routine is also wrong.

What the error does depends on your stack. A hardware controller may drop the drive out of the array and abort the rebuild, turning one dead disk into a dead volume. Linux md handling depends on the RAID level, kernel, and available copies; a read error can still make data unrecoverable. Which behaviour you have is a thing to know before the day it matters.

Then there is time. A 16 TB drive resilvering at a real-world 150 MB/s needs about thirty hours if nothing else touches the array. Production I/O and rebuild throttles routinely stretch that to two or three days. For every hour of it the array has zero redundancy, and every surviving disk is being read end to end at full load.

Disks bought together fail together. Large-scale field studies of production populations have found replacement rates several times the datasheet MTBF, and failures clustered in time rather than spread evenly across a fleet. The neighbour of a dead drive is the most likely next candidate, and the rebuild hands it the heaviest week of its life.

Parity knows something is wrong, not what

Scrub a RAID 5 array and it reads every stripe, recomputes parity, and counts mismatches. That count says the stripe is inconsistent. It cannot say whether the data block is wrong or the parity block is.

Linux md resolves the ambiguity by assumption. Writing repair to /sys/block/md0/md/sync_action recomputes parity from the data blocks and writes it back, which is right when parity drifted and quietly launders the evidence when the data drifted. RAID 1 repair selects one mirror copy without proving that it contains the correct data. Neither is a bug — there is no information available to do better.

The write hole is the same shortage of information in a different place. A stripe update touches a data block and a parity block. Power loss between the two leaves the stripe internally inconsistent, and nothing records that fact. Rebuild off that parity months later and you get reconstructed blocks that were never real. mdadm closes this with a write journal or a partial parity log; hardware controllers close it with a battery-backed cache, which works until the battery ages out unnoticed.

Checksummed filesystems remove the ambiguity by storing a hash with every block. ZFS and btrfs read a block, check the hash, and when it fails they can identify a valid mirror or reconstruction if an intact redundant copy exists. Without one, a checksum can detect corruption but cannot restore the data. That is the line between detecting inconsistency and correcting corruption, and it is the strongest argument for ZFS over a hardware card for data you intend to keep.

Whatever you run, schedule the scrub monthly and alert on the counter:

cat /sys/block/md0/md/mismatch_cnt

A nonzero result on a RAID 5 array is information you want on a quiet Tuesday, not halfway through a rebuild.

The controller is part of your on-disk format

Hardware RAID cards write their own metadata to the member disks: stripe geometry, member order, array state. The format is vendor-specific and mostly undocumented. When the card dies, the disks are simultaneously intact and unreadable, and recovery starts with sourcing the same card family on compatible firmware before anything can be mounted.

Software RAID avoids that particular dependency on a controller family. The Linux md superblock is documented, and a compatible Linux host can assemble the array. A ZFS pool can import on another compatible ZFS system, provided its implementation supports the pool's enabled features. For a small business with no spare controller on a shelf, that portability outweighs the throughput the card was bought for.

Whichever you run, save the geometry now, while everything works:

mdadm --detail --scan
for d in /dev/sd[b-i]; do mdadm --examine "$d"; done
lsblk -o NAME,SERIAL,SIZE,MODEL

Keep the output, the controller model and firmware version, and a photo of which serial number sits in which slot. Store it off the array. Rebuilding those parameters after a failure is one of the most expensive parts of a RAID recovery, and today it costs you five minutes.

Why the recovery quote is a multiple, not a markup

A single failed disk is one job. Image it, then work on the image — the rule zero of any recovery.

An array is that job N times before anyone opens a file. Every member gets imaged, because reading a degraded array live risks the disks you still have. Then the geometry has to be established from the images: chunk size, member order, parity rotation (left-symmetric is the Linux default, controllers vary), data offset, and which member had already gone stale before the array stopped.

Only then can stripes be reassembled, and the output is only as complete as the inputs. RAID 5 missing one member reconstructs from parity, and every unreadable sector on the survivors becomes a hole in a file. RAID 5 missing two members, or a RAID 0 stripe missing any member, is arithmetic without a solution.

The most expensive failures are not the disks. They are the decisions made in the first hour: rebuilding onto the wrong slot, accepting a controller prompt to initialize the array, running a filesystem repair against a degraded volume. Each one writes fresh data over the layout the recovery depends on. The correct first move on a failed array is to power it down and photograph the slot order.

What to run instead

Pick the RAID level for the rebuild, not the capacity. Above roughly 4 TB per drive, RAID 6 or a suitably arranged RAID 10 can retain more fault tolerance during a rebuild than RAID 5. RAID 10's protection depends on which mirror members fail. ZFS mirrored vdevs resilver only allocated data, which finishes sooner and reads the survivors less.

Mix your batches and keep a hot spare. Different purchase dates or different vendors break the correlation. A hot spare starts the rebuild at 3am instead of when someone reads the email.

Monitor the array, not the server. SMART attributes, mdadm --monitor or zpool status, and an alert when the array goes degraded. A degraded array nobody noticed is the standard prologue to a total loss, and it belongs in the same minimal monitoring setup as your uptime checks.

Run 3-2-1. Three copies, on two media types, with one offsite. The rule comes from photographer Peter Krogh's digital asset management work and has outlived every storage generation since. The current variant adds one copy offline or immutable, and zero errors on verification.

Verify the restore, not the job status. A backup that has never been restored is an untested assumption with a green tick next to it. Restore on a schedule, to a separate machine, and check the contents — the same discipline that makes a pg_dump into an actual backup.

Write the array-failure runbook before you need it. Power down, do not initialize, photograph the slots, image every member. Keep these steps visible and test the recovery procedure before an incident.

The short version

RAID answers "can we keep working while a disk is dead". Backup answers "can we get the data back". They are different questions, and buying an answer to the first has never been an answer to the second.

Size the array for its rebuild, put checksums under it, scrub it on a schedule, and keep a copy somewhere the fire cannot reach. Then test the restore, because the only backup that counts is one you have already put back.