Panic Buttons at Scale: Designing an Emergency Alert Fleet for 100 Locations

Panic Buttons at Scale: Designing an Emergency Alert Fleet for 100 Locations

At fleet scale, the hard part of a panic button isn't sending an alert. It's knowing every button will work on the day someone's safety depends on it.

A panic button demo is a weekend project: a button, a network, a message in a group chat. The demo convinces everyone, the pilot succeeds, and then the request arrives — "roll this out to a hundred locations." That's where the actual product begins.

From proof of concept to fleet

The case behind this article is a security-operations deployment that grew from a single-site proof of concept to roughly a hundred locations. The PoC answered one question: does a press produce an alert? The fleet forced the real ones.

Which of the hundred sites are alive right now? Who exactly gets the alert for site 37, and who gets it when they don't respond? How do monthly drills run without training everyone to ignore alarms? Every one of those questions became more engineering than the button itself.

Hardware and the connectivity fallback

Each site runs physical buttons wired to a small gateway. The gateway publishes button state upstream over MQTT — the same device-side protocol choice as any telemetry fleet, because an alarm is just telemetry with consequences.

Connectivity is where emergency systems earn distrust. Building WiFi fails — and it tends to fail during exactly the kinds of events that make people reach for a panic button. So every gateway carries a GSM fallback: primary path over the site network, cellular modem when it drops.

The rule that matters: the fallback gets exercised on a schedule. A failover path that has never carried traffic is a hope, not a design.

Supervised liveness: the real product

Picture a security guard required to check in with the front desk every 30 minutes. If he misses a check-in, his supervisor knows something's wrong immediately — not once an incident happens. That's supervised liveness: every gateway publishes a heartbeat on a fixed interval, a routine "I'm still alive" report.

Emergency systems fail silently. A dead button looks exactly like a calm day — no traffic either way. An unsupervised fleet decays invisibly until the one press that matters goes nowhere.

A background health worker walks the fleet: silent past one threshold and the site drops to warning; silent past a second and it's marked offline and the facilities workflow kicks in. In this deployment both thresholds sit under a minute, and both are configurable. A dead button gets discovered within minutes on an ordinary Tuesday — not during an emergency.

One design detail keeps the fleet quiet: heartbeats that merely repeat the current state never create incidents. Presence and alarms are separate channels. Mixing them buries the one message that matters under thousands that don't.

Alert routing and escalation

A press on an armed button becomes an emergency, routed by area and by role. On-site responders are notified first, over a channel people actually watch — in this fleet that's Telegram, delivered only to approved, pre-registered targets.

If nobody acknowledges within a set window, the alert escalates to the next ring: area supervisor, then the central operations desk.

The escalation chain is encoded, not hoped for. At 2 a.m. nobody improvises a phone tree — the system already knows ring two and ring three, and it walks them on a timer.

Test mode versus real alarms

A fleet this size runs drills — regulations and common sense both demand it. But paging the full escalation chain at every monthly drill is how you manufacture alarm fatigue. Alarm fatigue is the quiet killer of alerting systems: the hundredth ignored notification looks identical to the first real one.

So drills run in an explicit test mode. A test press exercises the entire pipeline — button, gateway, broker, routing — but lands in a drill channel and gets labeled as a test in the record. Real escalation stays reserved for real events, which is exactly what keeps it credible.

False triggers

Buttons get pressed by accident — cleaning staff, curious visitors, a box leaned against a wall. Two mechanisms keep that from eroding the system.

First, buttons carry an armed/disarmed state, so a press on a disarmed button is recorded but pages no one. Second, every incident gets an operator disposition — real, false, or drill — resolved from a dashboard scoped by role and area.

That disposition data is a fleet metric. A site logging false alarms every week damages trust more than a site that's offline — offline at least looks broken.

The audit trail

Every press, every state transition, every notification sent, every acknowledgment and resolution gets recorded append-only. After a real incident there are always questions: when was the press, who was notified, how fast was the response.

The log answers them; memory doesn't. The same trail proves drills actually happened and feeds the tuning of thresholds and escalation windows.

The takeaway

The button is five percent of a panic-button system. The rest is supervised liveness, routing, escalation, drill discipline, and an audit trail — the same MQTT-ingest-to-live-state structure as any IoT backend, with the stakes turned up.

Build the boring 95 percent first. That's the part that works at 2 a.m.