An incident is stressful, and a black box that says “thinking…” for ninety seconds makes it worse. We want you to watch the teammate work: the tokens as they form, the tool it just called, the log line it just pulled. And we want that view to survive you closing your laptop, switching networks, and coming back an hour later to an investigation that never stopped.
Those two goals, low-latency live streaming and durability across disconnects, usually pull against each other. Streaming wants to be fast and fire-and-forget; durability wants everything persisted and ordered. Here’s how we stopped them fighting.
The one-laptop version
Open a WebSocket, forward every token as it arrives, render on receipt. It works on your laptop with one user. Then you notice tokens arriving thirty times a second cause a re-render storm; that a single dropped frame corrupts the message because deltas are additive, not idempotent; that a second browser tab shows a different state; and that a reconnect either replays the whole message from scratch or loses it entirely. Every one of those is a production bug, not an edge case.
The design
Tokens are coalesced by a batcher that flushes on a 50 ms timer, so the socket sees at most ~20 emits/second no matter how fast the model talks. Every flush carries a monotonically increasing seq and a partId that changes at each new model completion, and the async publishes are threaded through a single serialized promise chain, publishChain = publishChain.then(next), so deltas can never reorder under concurrency. Critically, each flush does a dual write: the incremental delta goes to a Redis pub/sub channel (the live path), and the full accumulated text goes to a short-TTL Redis buffer keyed msg-stream:<id>, TTL 300s (the replay source).
On the client, deltas run through an idempotent, monotonic reducer that sits outside React Query entirely (via useSyncExternalStore), so token updates never invalidate or re-fetch anything:
// applyDelta, the client is allowed to drop or de-dup, never corrupt
if (partId !== cur.partId || seq === 1) reset(cur) // new stream
if (seq <= cur.seq) return // stale / duplicate → ignore
cur.text += delta; cur.seq = seqOn reconnect, the client re-subscribes to the thread room, pulls the accumulated snapshot with get_stream_snapshot, and reconciles against what it already has. Reconnection becomes a repair, not a restart. That works only because the agent is a durable workflow that never stopped running server-side (see post 3). The timeline of the investigation is replayed the same way, from a separate Redis list capped at the last 200 events.
Two Redis primitives, on purpose
The live token path is drop-tolerant pub/sub, fanned out across pods by a Socket.IO Redis adapter so an event produced on the pod running the agent reaches a socket held by any other pod, routed into rooms (thread:<id>, org:<id>, user:<id>). But the agent’s work queue (the tasks that must not be lost) runs on Redis Streams consumer groups with dedicated reclaim workers that pick up messages abandoned by a crashed consumer. Same datastore, opposite guarantees: at-most-once and cheap for the UI, at-least-once and durable for the work. Picking the wrong one for either job is a subtle, production-only failure.
The bug that taught us the pattern
The AI Overview feed lets you approve or decline an action inline. We shipped it, and watched approved actions flicker back into buttons a second later. The activity cache stored threads unbound, without the expensive per-request connector state, because binding costs three repository loads. A thread_updated socket event then merged an unbound thread over the bound one and erased the state the user had just set. The fix was to bind execution state late. On every socket merge we re-derive each action’s state from the merged data (a reattachThreadActionExecutionState step), so the cheap cached object stays cheap and the UI never regresses. You only find that class of bug by watching a real user’s click undo itself.
Build-it-yourself reality check
“Just stream the tokens” is one afternoon. Ordering guarantees under concurrent publishes, mid-stream join correctness, multi-tab and multi-pod fan-out, and snapshot-based reconnect are each their own afternoon, and the interactions between them are where the real time goes.
The deeper trap is treating streaming as a transport problem. It’s a state-synchronisation problem wearing a transport costume: the server holds authoritative, evolving state; N clients hold caches that drift; and the reconnect story is a distributed-systems reconciliation, not a socket reconnect. Get that framing wrong and you ship a feed that lies to users under exactly the conditions when they most need to trust it: flaky networks, mid-incident.
The hard part isn’t streaming tokens. It’s streaming them honestly, so a reconnecting user rejoins a running mind, not a torn one.
