Handling 100+ Camera Streams in One Go Backend

In this system of about 120 cameras, goroutine count was not the first limit I encountered. Recorder bandwidth and session limits mattered more, especially when connections recovered together after a power cut. Share connections for the same stream, back off between retries, and set live-view limits from measured capacity.

That is the summary of a recent build for a building with about 120 cameras spread across three network video recorders (NVRs, the boxes that store the footage) and a handful of operator PCs in the control room. The rest of this article covers where the real limits sit and the patterns the backend uses to handle camera connection failures.

What does the system look like?

The rule that shapes everything: only the server talks to cameras and recorders. Operator PCs talk to the server. Six operators watching the same lift camera and video profile can share one upstream pull from the recorder.

Two kinds of traffic leave the recorders. Video travels over RTSP, the standard streaming protocol cameras speak (we covered its quirks in why camera integration breaks). Events, such as "motion started on channel 12", come over a long-lived HTTP connection that the recorder keeps open and writes a line to whenever something changes.

The video relay is go2rtc, an open-source (MIT) streaming server written in Go. It pulls each stream, a specific camera and video profile, once and shares it with the browsers that request it. The Go backend decides which streams exist, who may watch them, and what happens when a camera disappears.

Is one background task per camera too many?

A goroutine lets work proceed concurrently in Go; that does not mean every task executes in parallel at the same instant. Go schedules goroutines onto operating-system threads. The Go FAQ says a new goroutine starts with a few kilobytes of stack, which can grow. That is not the total memory cost of a camera connection: buffers, protocol state, and network resources also count.

The better question is how many connections you open to the devices, because recorder connection limits can be reached before goroutine count becomes a constraint. Count them per layer:

The goroutine count roughly follows the "What we run" column. More importantly, so does the load on the recorders.

Where does it actually start to struggle?

The live-video path forwards video without decoding frames or re-encoding them. Recorder capacity and operator PC performance still need checking. An optional smooth-reverse path uses FFmpeg to decode and re-encode video. It is disabled by default in the reviewed configuration; ordinary reverse playback uses keyframes from the recorder. The optional path requires its feature flag, one NVR camera, −1× or −2× speed, and a ready encoder.

The specifications for the NVR model on this site list 400 Mbps of outgoing transmission and only one concurrent search/playback user per unit. The live limit reads "10 users", which turned out to mean ten client sessions: every channel opened with the same session key counts as one. In a field test on 29 September 2026, one NVR served 27 channels of 1080p video at once through a single session, with 26 delivering frames.

This test shows why bandwidth and session behaviour matter. It does not establish a limit for every NVR model. A few practical consequences:

• Use sub-streams for grids. A 5×5 grid of full-HD streams costs the recorder and the operator PC far more than it shows. Small tiles get the low-resolution profile; double-click opens the HD stream.
• Avoid transcoding unless it is needed. Decoding and re-encoding video requires additional processing. Pass-through relaying avoids that work but still consumes network capacity, CPU, and memory.
• Set limits from measured capacity. We cap live streams per NVR through a setting that defaults to "off", and turn it on only once a recorder is proven to struggle.

For storage-side math on the same kind of site, see sizing CCTV storage.

How do you reconnect without hammering the recorder?

Cameras drop. Switches reboot, power flickers, and someone unplugs a cable to charge a phone. After a power cut, forty cameras come back at the same moment. If every one retries every second, the recorder gets a flood of logins exactly when it is weakest.

This system uses backoff, media-activity checks, and a pause after repeated authentication failures. Adding jitter is a recommendation; the source loop reviewed here does not yet implement it. These are the patterns and their limits:

1. Exponential backoff. Wait longer after each failure: 3, 9, 27, 81 seconds, with the base wait capped at 5 minutes. In the example below, added jitter can bring the total wait close to 6 minutes. A dead camera then costs a knock every few minutes, not every few seconds.
2. Check media activity rather than connection duration alone. A dead camera behind a live recorder may accept a connection and send nothing until a timeout fires a minute later. A media-byte counter helps decide when to reset the wait to 3 seconds. In this implementation, it includes RTP and RTCP packets, so an increase does not prove frames were decoded or displayed. Check video packets or decoding separately for picture health.
3. Consider jitter, a small random offset to each wait. Marc Brooker's analysis on the AWS Architecture Blog (March 2015) showed that randomised backoff sharply reduces contention compared with plain exponential backoff. Without it, forty cameras that failed together retry together.
4. Stop after repeated password rejections. On the event connection, three consecutive HTTP 401 authentication failures after authentication handling pause that recorder for 15 minutes. An initial Digest authentication challenge does not count as a rejected password. Account lockout depends on the recorder model and configuration, and a locked admin account on site is worse than a short gap in events.

This simplified retry loop adds the recommended jitter. It uses math/rand/v2 (Go 1.22 or later) and is not a complete copy of production code. received counts media bytes; pull must handle context cancellation and close the connection. The 15-minute authentication pause and HTTP watchdog are implemented separately:

wait := 3 * time.Second
for ctx.Err() == nil {
	before := received.Load()
	err := pull(ctx, stream, &received) // returns when the connection ends
	if ctx.Err() != nil {
		return
	}
	if received.Load() > before { // media activity, not proof that frames displayed
		wait = 3 * time.Second
	}
	jitter := time.Duration(rand.Int64N(int64(wait / 5)))
	delay := wait + jitter
	log.Printf("%s: %v (retry in %s)", stream, err, delay)
	select {
	case <-ctx.Done():
		return
	case <-time.After(delay):
	}
	wait = min(wait*3, 5*time.Minute)
}

One more failure deserves a guard: the connection that dies silently. The event connection to a recorder can stay "open" while nothing flows, after a switch reboot for example. The watchdog in this system closes it after 10 minutes without a byte and reconnects. This is a recovery policy, not proof that the device is offline: a connection with no events can also be quiet. Set the threshold from the NVR model's heartbeat behaviour.

What happens when one client is slow?

Here backpressure matters: how a system behaves when a consumer cannot keep up with the producer. Think of a supermarket queue. One customer counting coins must not freeze every other till.

Motion events fan out to every connected operator PC. Each PC gets a small buffered channel (64 events). If that buffer is full, the event is dropped for that PC rather than blocking delivery to everyone else. This excerpt only shows the non-blocking send; subscriber access and channel lifetimes must be synchronized, as they are with a mutex in the source implementation:

for ch := range subscribers {
	select {
	case ch <- event:
	default: // slow client: drop for this one, never block the others
	}
}

This live-delivery path can lose events for a slow PC. Motion events are written to the database, but the reconnect path reviewed here only replays up to 200 recent events from memory. That history disappears on a server restart and does not guarantee complete recovery. A PC that stays connected after its buffer fills may also miss dropped events. If complete recovery is required, design persistent-history reads using sequence numbers and gap handling. Life-critical alarms have different delivery requirements, as described in nurse call and code blue alert design.

Two related guards stop bursts from the operator side:

• Reserve before you count. When an operator opens a 25-tile view, all 25 requests arrive within milliseconds, before the relay has registered any of them. A limit check that only reads "active streams" lets all 25 through. Each approved stream is therefore reserved for 15 seconds, so concurrent requests see each other.
• Hold streams briefly after the last viewer leaves. Operators flip between views constantly. Keeping a stream connected for one minute after its last viewer can avoid a fresh RTSP handshake when the operator returns. The picture may still take time to appear because of keyframes, transport, or browser rendering. The connection still uses network capacity, processing, and buffers during that minute. The benefit is avoiding a fresh connection when the operator returns.

How do you stop tasks that never finish?

A leaked goroutine is one waiting forever on something that will never happen: a channel nobody writes to, a connection nobody closes. With cameras being added, edited, and removed from an admin screen, every supervising loop needs an exit.

The pattern is one cancellable context per camera or per recorder. Editing a camera cancels its context. The reader explicitly closes the connection when it receives ctx.Done(), allowing the loop to stop. Context cancellation alone does not automatically close every network connection. A fresh loop then uses the new settings, and the old supervisor entry is cleaned up when its task finishes.

The reviewed reader still has a cleanup gap: each connection attempt starts a helper that waits for the camera context to end. If the connection fails earlier, closing the socket does not end that helper. Retries can accumulate waiting helpers until the camera context is cancelled. A connection-scoped context, cancelled when that attempt returns, would also stop its helper. This is a recommended fix, not a property the reviewed implementation already has.

Two Go changes help here:

• Go 1.26 (February 2026) adds an experimental goroutineleak profile that reports goroutines blocked on channels or mutexes that can never unblock. It is enabled at build time with GOEXPERIMENT=goroutineleakprofile. On Go 1.26, it can be tried in staging builds of long-running supervisors. It does not detect every kind of leak.
• Go 1.25 (August 2025) made GOMAXPROCS, the limit on logical CPUs executing Go code simultaneously, consider container CPU limits on Linux by default. A 2-CPU limit can bound parallelism even on a host with 16 logical CPUs. Manual GOMAXPROCS settings or module compatibility settings can disable this default behaviour.

Checklist before adding camera #121

Go helps manage concurrent work. Recovery after a power cut still depends on recorder capacity, reconnect control, and handling missed events. For camera health beyond video, such as uptime and port status, pair this with SNMP monitoring for CCTV.

If you run a camera or device network that has outgrown its software, let's talk about the backend it needs.