Postgres Maintenance Took Down CircleCI Twice

On 6 and 9 October 2026, CircleCI's job scheduling stopped because routine PostgreSQL cleanup wrote transaction logs faster than they could be archived, and the disk holding those logs filled up. No bug, no attack. If your business runs on Postgres, watch how fast the log disk fills and whether archiving keeps up, not just CPU and free space.

This article walks through CircleCI's two incident reports, explains in plain terms why a cleanup job can stop a database, and lists the checks that catch the same pattern on a much smaller system.

What happened on 6 October?

CircleCI's post-incident report for 6 October traces the outage to the database behind its orchestration service, the part that decides which job runs next. The build machines were healthy. They simply had nobody telling them what to run.

About 3,500 jobs failed to start and customers were told to rerun them. The report is candid that the warning signs were there: the team says it "should have recognized the combination of alerts sooner." Short alerts had fired and cleared on their own since the morning.

The move itself was the lesser evil. Had the log volume filled completely, Postgres would have stopped accepting writes. Moving the logs cost 50 minutes of downtime; doing nothing would have cost an open-ended read-only database.

Why did it happen again three days later?

The 9 October report describes a second database, the one behind Temporal, a workflow engine that remembers where every long-running process is up to. Temporal deletes the history of finished work after three days. The heavy traffic from 6 October therefore came due for deletion on 9 October, and those mass deletions produced the same flood of log writes.

The numbers are worth reading slowly:

• 08:00 UTC (15:00 WIB): free space on the log volume starts falling from 0.8 TB.
• 10:10 UTC: 0.1 TB left. Alerts fire about 20 minutes later.
• 11:15 UTC (18:15 WIB): the volume is full and Postgres goes read-only. All Temporal processing halts.
• The volume had a fixed size of 1 TB and 3,000 IOPS (disk operations per second), limits CircleCI calls hardcoded and impossible to raise.

The restart to move the logs brought the database back at 12:59 UTC, but the trouble didn't end there. Cleanup of dead rows had fallen behind by tens of millions, so every query waded through rubbish. Manual cleanup jobs competed with real queries for CPU and were cancelled. Upstream systems retried failed work all at once, a retry storm that kept the database pinned. Only after all incoming and retried work was paused at 20:48 UTC and throttled back in gradually was the incident mitigated, at 23:47 UTC (06:47 WIB, 10 October). End to end, roughly 12.5 hours of degraded service.

Why can a cleanup job stop a database?

Postgres never edits a row in place. An update writes a new version and leaves the old one behind, like a ledger where corrections are written on a new line and the old line is crossed out. A process called VACUUM later clears the crossed-out lines. Its automatic form, autovacuum, runs in the background.

There is a second, less visible job. Every transaction gets a number from a 32-bit counter, which means the counter eventually wraps around, roughly every two billion transactions. Before that happens, old rows must be "frozen", marked as permanently visible so the wrap doesn't make them look like they're from the future. The PostgreSQL 18 documentation on routine vacuuming sets the default trigger, autovacuum_freeze_max_age, at 200 million transactions. Past that age the database forces an anti-wraparound autovacuum on the table, even if autovacuum is disabled. If it still falls behind, warnings start when 40 million transactions remain, and below 3 million the database refuses any command that needs a new transaction number.

Freezing a huge table means rewriting a lot of pages, and every change is first recorded in the write-ahead log (WAL), a journal Postgres writes before touching the data so it can recover after a crash. When WAL archiving is on, which it is in any serious backup setup, the documentation on WAL configuration states that old log segments cannot be removed until they are archived. If archiving can't keep up, the files pile up in pg_wal.

Note what this chain does not contain: a bug. Every step is Postgres doing its job correctly. The failure lives in the gap between how fast logs are written and how fast they leave.

Does this happen on small systems too?

Yes, and usually for duller reasons. A hotel PMS, a point-of-sale backend or a self-hosted Supabase instance won't generate CircleCI's volume, but it has smaller disks and nobody watching them at night. The variants I see most often:

The third row is exactly the 9 October trigger. Data retention rules are good practice, but deleting millions of rows in one go writes as much log as inserting them did.

What should you monitor?

Disk usage alerts at 90% arrive too late. CircleCI went from 0.8 TB free to full in a little over three hours. The useful signal is the rate: free space divided by how fast it's shrinking gives you the time until the disk is full, and an alert at "less than six hours left" wakes someone while there's still room to act.

Four queries cover most of it:

-- 1. Is archiving failing? A growing failed_count or an old
--    last_archived_time means WAL is piling up.
SELECT archived_count, last_archived_time,
       failed_count, last_failed_time
FROM pg_stat_archiver;

-- 2. Which replication slots hold WAL back, and how much?
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;

-- 3. How close is each database to forced freezing? Compare with
--    autovacuum_freeze_max_age (200 million by default).
SELECT datname, age(datfrozenxid) AS xid_age
FROM pg_database
ORDER BY xid_age DESC;

-- 4. How much WAL is on disk right now?
SELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();

Feed these into whatever monitoring you already have. Choosing thresholds and routing alerts to someone who will act is covered in monitoring and alerting for small systems.

How do you design so it doesn't happen?

Give the log disk room to grow, or put it on the data volume. CircleCI's permanent fix for the Temporal database was to stop using a fixed-size log volume. A separate log disk is fine as long as it can be expanded online and has headroom for a full cleanup of your largest table.

Cap what a replication slot can hold. Set max_slot_wal_keep_size to a value your disk can absorb. A replica that falls too far behind will then need to be rebuilt, which is far cheaper than the primary going read-only.

Delete old data in small batches, or drop it by partition. Splitting a table into partitions, separate monthly drawers, lets you throw away a whole month with one quick command instead of deleting rows one by one. Where partitioning isn't practical, delete in batches of a few thousand rows with a pause between them, and avoid running it on the day after a traffic spike.

Make clients wait instead of hammering. Both reports list retry behaviour as a follow-up: systems gave up and failed jobs, or retried and buried the database. Retries need a growing delay and a ceiling, and a recovered database needs work fed back in gradually.

Test that archiving actually works. A WAL archive that has never been restored is a hope, not a backup. The restore drill in verifying Postgres backups also proves the archive is complete.

What not to do when the log disk fills

Never delete files from pg_wal by hand. Postgres needs them to recover, and removing the wrong segment can leave a database that won't start. The safe moves, roughly in order:

1. Find what is holding WAL back: a failing archive, a stale slot, or a vacuum in progress.
2. Fix the archive or drop the slot that is no longer needed.
3. Expand the disk if your platform allows it online.
4. Only then consider moving WAL to another volume, which, as CircleCI found, means downtime.

What to check this week

• Run the four queries above on every production Postgres you depend on.
• List every replication slot and confirm who owns it. Drop the orphans.
• Set max_slot_wal_keep_size and add an alert on hours-until-full for the WAL disk.
• Review scheduled delete jobs: how many rows they touch and when they run.
• Ask your vendor or hosting provider who receives the archive-failure alert at night.

CircleCI's database did what it was designed to do, and the business still lost hours twice in one week. Maintenance is part of production load, so plan disk, alerts and retries around it.