What mega-sale traffic taught me about Kafka consumers
Lag is a symptom. Notes on preparing consumers for 10× traffic days.
Every big sale event follows the same script: traffic ramps up, dashboards turn yellow, and somewhere a consumer group quietly falls behind. Here’s what I’ve learned about getting ahead of it. (Sample content — replace with your own story.)
1. Measure the per-thread rate, not the total
The most useful number in capacity planning is records consumed per second, per thread. Once you know it, the math is simple:
required_threads = peak_production_rate / per_thread_rate × headroomIf your topic has fewer partitions than required_threads, adding pods won’t help — partitions are the ceiling.
2. Lag is a symptom
A consumer that's falling behind is rarely slow at consuming. It's slow at whatever it calls next.
Look downstream first: the database, the cache, the HTTP dependency with a 2s timeout.
3. Rehearse
| Check | Why |
|---|---|
| Partition count ≥ planned threads | Parallelism ceiling |
max.poll.records tuned |
Avoid rebalance storms |
| Alerts on lag growth, not lag | Catch trends early |
Load-test with realistic payloads, then do it again a week before the event.1
Footnotes
-
Production traffic is always weirder than your test data. ↩