Designing a real-time messaging pipeline for millions of events

Every "real-time" feature looks trivial in a demo and terrifying in production. Here's how we took a messaging system from a few hundred events a minute to millions a day — without the 3am pages.

Start with the failure modes, not the happy path

Before writing a line of pipeline code, I list everything that can go wrong: a consumer falls behind, a broker restarts, a message arrives twice, a downstream service times out. The architecture is really just a set of answers to those questions — and if you design the happy path first, you end up bolting the answers on later.

The core decision was to treat the message log as the source of truth and make every consumer idempotent. Once that holds, most failures become "replay from offset N" instead of "lose data and panic."

Back-pressure is a feature, not a bug

When traffic spikes, the worst thing a pipeline can do is accept everything and quietly run out of memory. We made back-pressure explicit: consumers pull at their own pace, the log buffers, and we alert on consumer lag rather than on crashes. Slow is recoverable; out-of-memory is not.

“Make failure boring. The goal isn't zero incidents — it's incidents that resolve themselves.”

Measure the thing your users feel

End-to-end latency — produced-to-delivered — is the only number that matters to a user waiting for a message. We tracked p95 of that single metric and tuned everything else in service of it: batching, cache warming, connection pooling. Optimizing component metrics in isolation is how you get a fast system that still feels slow.

What I'd tell my past self

Keep the pipeline boring and the consumers dumb. Push complexity to the edges where it's easy to reason about, version your message schemas from day one, and never ship a consumer you can't safely replay. Everything else is tuning.