Why SD Cards Die in Edge Devices and How to Design Around It

Why SD Cards Die in Edge Devices and How to Design Around It

A fleet of Linux gateways deployed across a few dozen sites will start returning dead units somewhere between month twelve and month eighteen. The symptom is consistent: the device boots, the filesystem remounts read-only, and a technician swaps the card. Six months later the replacement does the same thing.

The card is not defective and the site is not unusually hot. The device was designed to write continuously to a storage medium that has a finite number of writes and no way to report how many are left. That is a design problem, and it is fixable at design time for almost no money.

Wear leveling only rotates the blocks you left empty

NAND flash cells survive a limited number of erase cycles. Wear leveling is the controller's job of spreading those erases so no single block dies early. The word covers two very different implementations, and the cheap one is what ships in a consumer microSD card.

Dynamic wear leveling rotates writes across blocks that are currently free. Static wear leveling also relocates cold data — your kernel, your binaries, files written once at provisioning and never touched — so those blocks join the rotation. Enterprise SSDs do static leveling. A five-dollar card usually does not.

The consequence decides the lifetime of your device. On an 8 GB card holding a 6 GB root filesystem, every write in the device's life lands on the remaining 2 GB. The other 6 GB of perfectly healthy NAND sits idle and absorbs nothing.

The write budget nobody calculates

Do the arithmetic before deployment, because it takes two minutes and changes the hardware decision.

Take a gateway writing 200-byte log lines five times a second. That is 1 KB/s, 86 MB/day, roughly 31 GB per year of writes as the host sees them. Against a 2 GB rotating pool that looks like fifteen full cycles per year — harmless.

The host's view is not what the NAND does. Write amplification is the ratio between bytes your application writes and bytes the flash actually programs, and on a cheap card with 4 MB erase blocks it is brutal. A 200-byte synchronous append cannot modify 200 bytes. The controller reads a whole block, merges the change, programs it elsewhere, and marks the original for erase. Add ext4 journal commits every five seconds, directory metadata, and mtime updates, and amplification for small synchronous writes commonly lands in the 10x to 100x range.

At 50x, that same gateway pushes about 1,500 GB per year through a 2 GB pool. Call it 750 erase cycles per year. Consumer 3D TLC is rated for roughly 1,000 cycles at the cell level. The card dies at around fifteen months, which is exactly when the field reports arrive.

Three numbers drive the outcome, and you control all three:

• Bytes written per day. Reduce by batching and by not logging at debug level in production.
• Size of the free pool. Increase by buying a larger card and partitioning only part of it.
• Cell endurance. Buy an industrial card rated in TBW instead of a retail card rated in nothing.

Power loss destroys data you wrote months ago

Wear is the slow failure. Power loss is the fast one, and it damages more than the file that was open.

The flash translation layer maintains a map from logical block addresses to physical NAND pages, and that map itself lives in flash. Cut power while the controller is programming a page and updating the map, and a card without power-loss protection can lose the whole erase block — including data written long before this boot. Your filesystem journal was never consulted, because the block device returned success and then lost the data anyway.

This is why an ext4 journal is not the answer people think it is. The default data=ordered mode guarantees filesystem metadata consistency after a crash. It guarantees nothing about a block device that acknowledges a write and discards it. FAT32, still the default on many vendor images, does not even have the journal: a power cut during a FAT table update can orphan half the volume, which is the failure mode behind most field recovery jobs on edge storage.

Industrial cards with onboard capacitors hold enough charge to finish the in-flight program operation and flush the mapping table. That capacitor is most of what you are paying for.

Design: move the writes off the card

The reliable pattern is boring. Make the card read-only in normal operation and route every write to memory or to the network.

Read-only rootfs with an overlay. Mount the root filesystem read-only and put a tmpfs overlay on top. Writes land in RAM and vanish on reboot, which is what you want for a device whose state lives on the server anyway. On Raspberry Pi OS this is a raspi-config option; on a custom image it is an overlayroot or a small initramfs hook. Any process that insists on writing to /var keeps working and simply writes to RAM.

Logs to tmpfs, then off-device. Set Storage=volatile and a RuntimeMaxUse cap in journald.conf, then forward to a central log server. Sizing the tmpfs is the whole trick: too small and you lose the context you need for debugging, too large and a chatty process triggers the OOM killer. 64 MB is a reasonable start on a 1 GB device.

Batch telemetry instead of appending. A sensor loop that appends one row per reading with a sync after each is the worst possible pattern for flash. Buffer in memory, write once per minute, and accept that a crash loses sixty seconds of readings. If the data cannot be lost, spool to a file you rewrite whole rather than a file you append to constantly — the same reason camera-based meter readers upload frames and keep no local history.

Mount options for the partitions that stay writable. noatime removes a metadata write on every read. commit=60 on ext4 batches journal commits into one minute instead of five seconds, trading a wider data-loss window for far fewer erases. Both are one-line changes in /etc/fstab.

Overprovision by partitioning short. Buy a 32 GB card, create an 8 GB partition, and leave the rest unallocated. Blocks that were never written stay in the controller's free pool and quadruple the rotation space. Run blkdiscard on the whole device before partitioning so the controller knows the space is free — this is the cheapest single change on the list.

What to buy when the card has to stay

Not every design can move to eMMC, and eMMC soldered to a board you do not control is its own kind of trap. When a removable card is the constraint, the spec sheet has three things worth reading.

pSLC mode. Industrial cards run 3D TLC NAND in pseudo-SLC mode, storing one bit per cell instead of three. Capacity drops to about a third and endurance rises by roughly 20x — from around 1,000 cycles to 20,000 or more. An 8 GB pSLC card and a 32 GB TLC card can be built from identical silicon.

A TBW or P/E cycle rating that exists. Retail cards advertise speed class and A2 IOPS ratings, which describe throughput, not lifetime. If the datasheet does not state terabytes written or program/erase cycles, the vendor has not committed to a number and you cannot budget against one.

Power-loss protection. Named explicitly in the datasheet, usually as onboard capacitors or a power-fail-safe FTL.

The price gap looks bad in a bill of materials and disappears in the field. An industrial card costs a few times a retail one. A technician driving to a site to swap a card costs more than the difference, once.

Health checks that warn you before the truck roll

SD cards have no SMART. That is the honest starting point — you cannot query remaining life on a standard SD card the way you can on an SSD. Three signals get you most of the way anyway.

eMMC health, when you have eMMC. The EXT_CSD register carries PRE_EOL_INFO and DEVICE_LIFE_TIME_EST_TYP_A/B, and mmc-utils reads them:

mmc extcsd read /dev/mmcblk0 | grep -iE 'PRE_EOL|LIFE_TIME'

PRE_EOL_INFO of 0x02 means the reserved block pool is 80% consumed; 0x03 means 90%. Life-time estimates come in 10% increments. Poll weekly and alert on the transition — it is the only real early warning in the category.

Read-only remounts. When the kernel hits an I/O error it remounts the filesystem read-only, and the device usually keeps running in a degraded state nobody notices until someone needs it. Check mount output for ro on your data partition every few minutes and alert immediately. This single check turns silent failures into a scheduled visit, and it belongs in the same minimal monitoring setup as your uptime probes.

A canary file. Write a known 1 MB pattern at provisioning, read it back weekly, compare checksums. Cheap, catches silent corruption, and gives you evidence when a vendor claims the card is fine.

Kernel messages are the fourth signal and cost nothing to grep for. mmc0: error -110, I/O error, and EXT4-fs error in dmesg all mean the card is on the way out. Forward them with your logs and match on them centrally.

The short version

Assume the card will wear out and design so that it does not matter. Read-only rootfs, logs in RAM and shipped off-device, telemetry batched rather than appended, a partition sized well below the card's capacity, and an industrial card with power-loss protection where the budget allows.

Then instrument the failure you did not prevent. A device that reports a read-only remount on Tuesday is a maintenance ticket. The same device discovered dead in March, holding three months of data nobody can read, is an incident.