How Kafka Consumers Actually Work — Offsets, Consumer Groups, and Rebalancing
You've got 6 partitions and 100 million events a day. One consumer is falling behind. You spin up a second — now you've got duplicates. Kafka solves this, but the solution changes how you think about consuming data entirely.
In this episode, we build the complete Kafka data flow from scratch: how producers write without caring who's reading, how consumers track their position with offsets, and how consumer groups split work across multiple consumers without duplication.
We also cover one of Kafka's most painful operational realities — the rebalance — and what happens when a flaky consumer turns it into a storm that grinds your entire pipeline to a halt.
Timestamps:
0:00 — The problem: one consumer can't keep up, two consumers duplicate
1:30 — How producers write to partitions (and why they don't care about consumers)
3:00 — Offsets: the bookmark that drives everything on the consumer side
4:30 — Auto-commit vs manual commit — and why auto-commit can silently lose messages
6:30 — The replay superpower: resetting offsets to reprocess historical data
8:00 — Consumer groups: how Kafka splits partitions without duplication
9:30 — The parallelism ceiling: why partition count is your scaling limit
11:00 — Rebalancing: the stop-the-world event you need to understand
12:30 — Rebalance storms and how to avoid them
By the end of this video, you'll understand why offsets are the key to everything on the consumer side — independent progress, crash recovery, and replay — and why consumer groups give you horizontal scaling with a real operational cost.
📌 Kafka Under The Hood Playlist: [ • Kafka Internals ]
Previous episodes:
▶ Episode 1 —[ • Kafka Internals #1 — What Problem Does Kaf... ]
▶ Episode 2 — [ • How Kafka Scales: Partitions, Keys & the H... ]
Next up: What does a partition actually look like on disk? How does Kafka store millions of messages and still serve reads in milliseconds? The answer starts with how your hard drive works.
#Kafka #ApacheKafka #DistributedSystems #SystemDesign #BackendEngineering #SoftwareEngineering #KafkaConsumers #EventStreaming #UnderTheHoodEngineering