Clock Drift: When Timestamps Stop Being Evidence

Clock Drift: When Timestamps Stop Being Evidence

A timestamp is only evidence if the clock that wrote it was correct. In field deployments it usually isn't. Cameras, access controllers, and battery-powered sensors each keep their own time, and each drifts on its own schedule. When an incident forces you to line those logs up against one another, the drift is what you end up reading.

Nobody notices until they need to. A dashboard renders whatever time a device reports, and it looks fine. The bill comes due on the day somebody asks what happened at 14:32.

Why do field clocks drift at all?

Every device that keeps time across reboots has a real-time clock: a small oscillator, usually a 32.768 kHz quartz crystal, feeding ticks into a counter. Consumer-grade parts are specified around ±20 ppm at room temperature. That sounds like nothing until you multiply it out.

Twenty parts per million across a day of 86,400 seconds is 1.7 seconds. Call it 52 seconds a month — and that's the good case, on a device sitting in an air-conditioned rack.

Field devices don't sit in air-conditioned racks. Tuning-fork crystals follow a parabolic frequency curve peaking near 25 °C, with a curvature constant of roughly −0.034 ppm/°C² (Microchip AN2648). A controller baking at 50 °C inside a rooftop enclosure picks up about another 21 ppm of error on top of its rated tolerance — near 1.8 extra seconds lost per day. And it only loses. The parabola has no positive side.

Then there are the devices with no crystal at all. Espressif puts the ESP32's internal RC oscillator in the category where "time drifting in the range of several seconds per day is acceptable", and recommends the external 32 kHz crystal whenever accurate timekeeping through deep sleep matters (ESP-IDF System Time). On a battery sensor that wakes once an hour and syncs with nothing, several seconds a day becomes minutes inside two weeks.

DVRs and NVRs have their own version of this. The clock gets set by hand at commissioning, backed by a coin cell, and never touched again. When that cell dies — and after five years it has — the next power cut drops the clock to whatever default the firmware carries. The recorder keeps recording. The overlay keeps burning a timestamp into every frame. It's just wrong now.

What does a four-minute offset actually cost?

A building operator needed to settle a dispute about who entered a restricted corridor. The access controller had a badge event with a name and a time. The NVR had continuous footage. Two systems, one question, and it should have taken ten minutes.

Playback at the badge time showed an empty corridor. Nothing at the minute before, nothing at the minute after. The controller was domain-joined and synced. The NVR had been running on its own RTC since installation, and had gained a little under four minutes. The event was in the recording all along, sitting four minutes earlier than anyone was looking.

That is the cheap version of this failure, because it got caught. The expensive version is the one where nobody checks, the offset happens to be small enough to look plausible, and a reconstructed sequence of events puts two things in the wrong order.

Cross-system correlation is the whole point of keeping logs on separate systems. It only works if you know each system's offset from a common reference:

The normalization step is only possible if somebody measured the offset before the device was rebooted, re-synced, or replaced. After that, the number is gone and the old timestamps can never be placed on a shared timeline again.

Is running NTP enough?

NTP fixes the drift problem. It does not fix the trust problem, and in field deployments it often doesn't even get to fix the drift problem.

Three failures account for most of what I find on site:

• Sync at boot only. Plenty of embedded firmware calls SNTP once during startup and never again. A gateway with 90 days of uptime has 90 days of accumulated drift, and its config page still proudly shows an NTP server.
• UDP 123 blocked outbound. Camera and controller VLANs are usually isolated on purpose, and time is what gets forgotten in the ACL. The device is configured for NTP. It has never once reached a server.
• Nobody monitors the offset. Sync status is a metric like any other. chronyc tracking gives you the current offset in seconds; export it, chart it, alert when it exceeds a threshold you picked deliberately. This belongs in the same fleet health view as reachability and uptime — see monitoring CCTV fleets with SNMP for how that polling layer gets built.

There's a security dimension too, and it's less settled than most people assume. At a February 2026 talk covered by LWN, Ruben Nijveld described NTP as "fundamentally a broken protocol" on the grounds that a network attacker can modify timestamps in transit without detection. Network Time Security, standardized as RFC 8915 in 2020, fixes that with authenticated time. Uptake has been slow: the reference ntpd doesn't support NTS at all, and the same report notes no known SNTP implementations support it either. SNTP is exactly what embedded devices run. The gear where forged time would be hardest to notice is the gear that can't yet be protected.

For deployments in Indonesia, BMKG operates the national time reference at ntp.bmkg.go.id under Law 31/2009, alongside time.bmkg.go.id and jam.bmkg.go.id (BMKG Tanda Waktu). A local stratum source beats a random pool member on round-trip latency, and it gives you a defensible answer when someone asks which clock you trusted.

How do you prove a clock was right?

Digital forensics has a formal answer to this. In Forensic Science International: Digital Investigation volume 49 (2024), Vanini, Hargreaves, van Beek and Breitinger define time anchors: events whose real time is fixed by an external source you trust, which let an examiner test the device's own clock against reality and measure the skew instead of assuming it away.

The operational version of that idea is four habits:

1. Measure the offset before you touch anything. chronyc tracking, ntpdate -q pool.ntp.org, or a photograph of the NVR's on-screen clock next to a phone showing network time. Write the number into the incident record. Thirty seconds of work, and it's unrecoverable once the device reboots.
2. Prefer devices that log their own sync events. chrony's measurement log and systemd-timesyncd both leave a trail. A recorded step of 228 seconds is a direct statement that every timestamp before it was 228 seconds off.
3. Keep one externally verifiable anchor per system. An SMS delivery receipt, a payment gateway callback, an HTTP Date header from a cloud API. Something a third party can confirm independently of your infrastructure.
4. Never rewrite historical timestamps. Store the measured offset next to the data and correct at query time. Overwriting the record destroys the one thing that showed the record was ever wrong.

This is the same discipline that makes badge events and video usable together in the first place, covered from the integration side in VMS and access control integration.

What changes when leap seconds go away?

The rules underneath all of this are being rewritten right now. CGPM Resolution 4 (2022) states that the maximum value for the UT1−UTC difference "will be increased in, or before, 2035", which ends leap-second insertion in practice; the CIPM was asked to propose the new limit ahead of the 28th CGPM (BIPM). That meeting is this year.

At the same time Earth started spinning faster. Duncan Agnew's paper in Nature (27 March 2024) projects that a negative leap second — a second removed rather than added — may be required around 2029 (Nature). Every one of the 27 leap seconds inserted between 1972 and 2016 was positive. No production code anywhere has handled the other direction.

Four rules keep a system out of the way of all that:

• Store UTC. Not local time, not "whatever the device reported". A fleet spread across WIB, WITA and WIT renders one instant three ways; that's a display concern, not a storage format.
• Use a monotonic clock for durations. CLOCK_MONOTONIC, performance.now(), steady_clock. A watchdog that measures elapsed time with the wall clock reports a negative interval the moment NTP steps backwards.
• Slew rather than step on running systems. chrony's maxslewrate keeps ordering intact; reserve makestep for boot, when nothing is timing anything yet.
• Record the sync source in the log line. When two systems disagree later, the question is which one to believe, and the answer is in whatever they were each synced to.

A field checklist

• Every device that writes a timestamp has an NTP server configured and a proven path to reach it. Verify the path, not the config field.
• Sync runs as a daemon, not a one-shot at boot.
• Clock offset is a monitored metric with an alert threshold.
• RTC battery replacement is on the maintenance schedule with everything else.
• Battery-powered nodes with no crystal report their own uncertainty, or send a sequence number the server timestamps on arrival.
• Incident procedure starts with recording the offset of every device involved.

The takeaway

Timestamps look like facts. They're measurements, made by cheap oscillators in hot enclosures, and they degrade the same way any other uncalibrated instrument degrades.

The fix costs almost nothing while the system is healthy: a reachable time source, a daemon that keeps running, one metric on a dashboard. The cost of skipping it lands entirely on the day you need the logs to agree — and by then the offset that would have explained everything has already been rebooted away.