Why SD Cards Fail in Field Devices

A fleet of Linux gateways can start returning failed units after twelve to eighteen months when sustained writes exhaust the cards' endurance. The symptom is consistent: the device boots, the filesystem remounts read-only, and a technician swaps the card. Six months later the replacement does the same thing.

Card defects and heat need investigation too, but sustained writes are a design risk worth checking first. The device was designed to write continuously to a storage medium that has a finite number of writes and no way to report how many are left. That is a design problem, and it is fixable at design time for almost no money.

Wear leveling depends on the controller

NAND flash cells survive a limited number of erase cycles. Wear leveling is the controller's job of spreading those erases so no single block dies early. The term covers two different approaches; which one is used depends on the card controller. Verify the selected model.

Dynamic wear leveling rotates writes across blocks that are currently free. Static wear leveling also relocates cold data — your kernel, your binaries, files written once at provisioning and never touched — so those blocks join the rotation. Which approach a card uses is model-specific; price alone does not establish it.

With dynamic-only wear leveling, an 8 GB card holding 6 GB of static data may concentrate changing data in the remaining space. Static wear leveling can rotate cold data too. The controller’s actual behavior determines whether this simplified 2 GB rotation model applies.

The write budget nobody calculates

Do the arithmetic before deployment, because it takes two minutes and changes the hardware decision.

Take a gateway writing 200-byte log lines five times a second. That is 1 KB/s, 86 MB/day, roughly 31 GB per year of writes as the host sees them. Against a 2 GB rotating pool that looks like fifteen full cycles per year — harmless.

The host’s write volume is not the NAND write volume. Write amplification is the ratio between bytes the application writes and bytes the flash programs. Small synchronous writes can add filesystem metadata, journal, and controller work. Flash controllers may remap pages instead of rewriting an entire erase block for each append, so measure the workload rather than assuming a universal 10x to 100x multiplier.

As a hypothetical calculation using 50x amplification, that same gateway pushes about 1,500 GB per year through a 2 GB pool. Call it 750 erase cycles per year. Assume a 1,000-cycle cell-level endurance budget for this example; the actual rating must come from the card’s specification. Under those assumptions, the endurance budget is exhausted at around fifteen months. Actual lifetime depends on the card controller and workload.

Three numbers drive the outcome, and you control all three:

• Bytes written per day. Reduce by batching and by not logging at debug level in production.
• Size of the free pool. Increase by buying a larger card and partitioning only part of it.
• Cell endurance. Buy an industrial card rated in TBW instead of a retail card rated in nothing.

Power loss destroys data you wrote months ago

Wear is the slow failure. Power loss is the fast one, and it damages more than the file that was open.

The flash translation layer maintains a map from logical block addresses to physical NAND pages, and that map itself lives in flash. Cut power while the controller is programming a page and updating the map, and a card without power-loss protection can lose the whole erase block — including data written long before this boot. Your filesystem journal was never consulted, because the block device returned success and then lost the data anyway.

This is why an ext4 journal is not the answer people think it is. The default data=ordered mode guarantees filesystem metadata consistency after a crash. It guarantees nothing about a block device that acknowledges a write and discards it. FAT32, still the default on many vendor images, does not even have the journal: a power cut during a FAT table update can orphan half the volume, which is the failure mode behind most field recovery jobs on edge storage.

Industrial cards differ in power-loss protection. Check the exact model’s datasheet for controller recovery behavior and hardware protection; an industrial label does not prove the card contains capacitors.

Design: move the writes off the card

The reliable pattern is boring. Make the card read-only in normal operation and route every write to memory or to the network.

Read-only rootfs with an overlay. Mount the root filesystem read-only and put a tmpfs overlay on top. Writes land in RAM and vanish on reboot, which is what you want for a device whose state lives on the server anyway. On Raspberry Pi OS this is a raspi-config option; on a custom image it is an overlayroot or a small initramfs hook. Any process that insists on writing to /var keeps working and simply writes to RAM.

Logs to tmpfs, then off-device. Set Storage=volatile and a RuntimeMaxUse cap in journald.conf, then forward to a central log server. Sizing the tmpfs is the whole trick: too small and you lose the context you need for debugging, too large and a chatty process triggers the OOM killer. 64 MB is a reasonable start on a 1 GB device.

Batch telemetry instead of appending. A sensor loop that appends one row per reading with a sync after each is the worst possible pattern for flash. Buffer in memory, write once per minute, and accept that a crash loses sixty seconds of readings. If those readings cannot be lost, use durable local storage and a tested flush strategy before acknowledging them. Rewriting a whole spool file does not by itself guarantee durability or lower wear. Camera-based meter readers that upload frames and keep no local history make a different tradeoff: they avoid local retention and depend on successful upload.

Mount options for the partitions that stay writable. noatime removes a metadata write on every read. commit=60 on ext4 batches journal commits into one minute instead of five seconds, trading a wider data-loss window for fewer periodic journal commits. It does not override application fsync calls. Both are one-line changes in /etc/fstab.

Overprovision by partitioning short. Buy a 32 GB card, create an 8 GB partition, and leave the rest unallocated. Blocks that were never written stay in the controller's free pool and quadruple the rotation space. On a supported, unmounted card with no data to retain, blkdiscard can tell the controller that existing blocks are no longer needed. It discards the device's contents; controller behavior and overprovisioning benefits must be checked for the card in use.

What to buy when the card has to stay

Not every design can move to eMMC, and eMMC soldered to a board you do not control is its own kind of trap. When a removable card is the constraint, the spec sheet has three things worth reading.

pSLC mode. Some industrial cards store one bit per cell in pseudo-SLC mode to trade capacity for endurance. Compare the exact model’s cell rating and TBW with TLC alternatives; the industrial label alone does not imply pSLC or a universal 20x improvement.

A TBW or P/E cycle rating that exists. Retail cards advertise speed class and A2 IOPS ratings, which describe throughput, not lifetime. If the datasheet does not state terabytes written or program/erase cycles, the vendor has not committed to a number and you cannot budget against one.

Power-loss protection. Named explicitly in the datasheet, usually as onboard capacitors or a power-fail-safe FTL.

The price gap looks bad in a bill of materials and disappears in the field. An industrial card costs a few times a retail one. A technician driving to a site to swap a card costs more than the difference, once.

Health checks that warn you before the truck roll

SD cards have no SMART. That is the honest starting point — you cannot query remaining life on a standard SD card the way you can on an SSD. Three signals get you most of the way anyway.

eMMC health, when you have eMMC. The EXT_CSD register carries PRE_EOL_INFO and DEVICE_LIFE_TIME_EST_TYP_A/B, and mmc-utils reads them:

mmc extcsd read /dev/mmcblk0 | grep -iE 'PRE_EOL|LIFE_TIME'

PRE_EOL_INFO of 0x02 means the reserved block pool is 80% consumed; 0x03 means 90%. Life-time estimates come in 10% increments. Poll weekly and alert on the transition — this is a standardized health signal. Some industrial SD cards also provide vendor-specific lifetime monitoring.

Read-only remounts. When the kernel hits an I/O error it remounts the filesystem read-only, and the device usually keeps running in a degraded state nobody notices until someone needs it. Check mount output for ro on your data partition every few minutes and alert immediately. This single check turns silent failures into a scheduled visit, and it belongs in the same minimal monitoring setup as your uptime probes.

A canary file. Write a known 1 MB pattern at provisioning, read it back weekly, compare checksums. Cheap, catches silent corruption, and gives you evidence when a vendor claims the card is fine.

Kernel messages are the fourth signal and cost nothing to grep for. mmc0: error -110, I/O error, and EXT4-fs error in dmesg can indicate media, power, connection, or filesystem problems. Investigate the cause rather than treating these messages as proof of wear. Forward them with your logs and match on them centrally.

The short version

Assume the card will wear out and design so that it does not matter. Read-only rootfs, logs in RAM and shipped off-device, telemetry batched rather than appended, a partition sized well below the card's capacity, and an industrial card with power-loss protection where the budget allows.

Then instrument the failure you did not prevent. A device that reports a read-only remount on Tuesday is a maintenance ticket. The same device discovered dead in March, holding three months of data nobody can read, is an incident.