A 16-Year-Old SQLite Bug and the Limits of Boring Technology

A 16-Year-Old SQLite Bug and the Limits of Boring Technology

A bug hid inside SQLite for sixteen years. It silently deleted committed rows of data, with no error and no crash. The cause wasn't bad software — Tailscale was using SQLite in a way that drifted just slightly from how most other users run it.

In August 2026, Tailscale published a post-mortem six months in the making. It took that long to work out what was silently deleting rows from its databases. Nineteen separate corruption incidents, no consistent trigger, no error message. The bug wasn't in Tailscale's code. It was inside SQLite itself, bundled into more devices and applications than almost anything else, sitting there unnoticed for roughly sixteen years.

The write-up reached 1,221 points and 239 comments on Hacker News. SQLite sits somewhere in most engineers' own stacks, and the mechanism raises an obvious question: does this apply to us?

What actually broke

Think of a courier marking a package "delivered" the moment they set it on the truck, before it's actually dropped off. If the truck takes a wrong turn at exactly the wrong second, the system already says done — while the package is still in transit. This bug works almost exactly like that.

SQLite's default write-ahead logging (WAL) mode doesn't write directly to the main database file. New pages go into a separate WAL file first; a "checkpoint" later copies them into the main file and resets the log. That step is where the bug lived: a data race between the checkpoint copying WAL pages and a write transaction committing new ones.

When the timing lines up badly, the checkpoint concludes it copied every page, when in fact one or more committed pages never made it across. The transaction returned success. The data is gone.

The trigger is specific: a WAL-mode database with two or more connections — separate threads or processes — writing or checkpointing the same file at the same moment. SQLite's documentation confirms this is a data race with tight timing constraints. Miss that combination and you'd never see it. Hit it often enough, at Tailscale's scale, and you get nineteen incidents in six months.

Why sixteen years of production use didn't catch it

SQLite's own documentation states the bug was likely present from version 3.7.0, released July 2010, through 3.51.2, released January 2026 — fixed only starting with 3.51.3. Sixteen years, in probably the single most-deployed database engine in the world, and nobody found it.

That's not a mark against SQLite's engineering. It's a fact about what "battle-tested" actually proves. The overwhelming majority of deployments use one connection per process, checkpointing on SQLite's own schedule, well clear of the race window.

Tailscale's pattern was different: aggressive, manual checkpointing across many concurrent connections, squeezing more throughput out of SQLite than the defaults were built for. That's exactly where "everyone else already found the bugs" stops being true, because nobody else was doing quite the same thing.

Antithesis, a deterministic-testing company, reproduced the bug in about fifteen minutes with a generic concurrent write-and-checkpoint workload — one SQLite's own developers had never triggered organically. Not an exotic edge case. An ordinary workload shape almost nobody happened to construct, until someone deliberately did.

The real lesson isn't about SQLite

The advice to prefer boring, mature technology over the new hotness is still correct. I give it constantly. But "boring" is a claim about a specific usage pattern, not the software in the abstract.

A dependency earns its reputation from the sum of everyone else's usage, and that reputation transfers to you only as far as your usage overlaps with theirs. Step outside the default assumptions and you've quietly opted out of the testing that made the choice feel safe.

I see this most often in embedded and edge systems, where a team pushes a familiar tool into a role its defaults never anticipated: heavier concurrency, tighter timing, unusual recovery paths. Every one of those deviations is invisible in code review — it just looks like normal use of a trusted tool.

Silent corruption is the worst failure mode

Here's what should worry you more than the bug itself: nothing about it was loud. No crash, no exception, no log line. The write returned success, the checkpoint completed without complaint, and the only symptom was data that should have existed and didn't — discovered later by someone noticing a row was missing.

A database that refuses a write, or crashes on a bad page, is unpleasant but honest. It tells you something is wrong the moment it goes wrong. One that reports success while quietly dropping data gives you nothing to react to.

For an edge deployment with no DBA watching a dashboard, that gap between "looks fine" and "is fine" can run for months — the way it ran for six at Tailscale, a company that was specifically looking for it.

What this means for SQLite at the edge

None of this makes SQLite a bad choice for IoT, embedded, or edge deployments. It remains right for local, single-process, single-writer workloads, which describes most of them.

But the line around WAL mode's assumptions is sharper now. SQLite's documentation is explicit that WAL mode doesn't work over a network filesystem, because it requires every process touching the database to share memory on the same host. A database file on networked storage was already the wrong architecture before this bug.

The narrower risk is concurrent, multi-connection writing with manual checkpoint control — exactly Tailscale's pattern. Single-writer discipline — one process, one connection, default checkpoint schedule — keeps you outside the race this bug needed. Multiple processes hammering the same WAL file, checkpoints tuned by hand, is the deviation worth auditing now, on any version older than 3.51.3.

Cheap corruption detection every team should already have

You don't need to reproduce a sixteen-year-old race condition to guard against its category of failure. A few cheap habits catch silent corruption before it becomes a mystery.

• Run PRAGMA integrity_check (or the faster quick_check) on a schedule. It performs a full low-level consistency scan — misformatted records, missing pages, broken indexes — and returns a single ok row if nothing is wrong. Almost free to schedule, and it catches exactly this class of problem.
• Verify backups by restoring them. A backup you've never restored is a hope, not a plan. I cover this for Postgres backup verification; it applies just as directly to a SQLite file on an embedded device.
• Monitor for silent divergence, alongside uptime. Row counts or checksums against a secondary source surface missing data faster than any support ticket.

None of this needs new tooling, just the decision to give data integrity the same attention as latency and error rate.

Takeaway for engineering leaders

The dependency you audited hardest was probably the newest one. The one you never re-examined is the one installed years ago, left alone precisely because it's "boring" and has "always worked."

That's backwards. Scrutiny should scale with how far your usage has drifted from a dependency's defaults, not how new it is.

When I take over a system for a client — the same exercise as stabilizing an inherited codebase — one of the first questions is which battle-tested pieces are used in an unusual way. Not because the tools are suspect. Because "battle-tested" only protects you from battles other people already fought the same way you're fighting them.