How Kafka Scales: Partitions, Keys & the Hot Partition Problem Explained
What happens when a single Kafka log can't keep up with 100,000 messages per second? You split it.
In this episode, we break down how Kafka partitions work — why they exist, how messages get routed to them, and what happens when you pick the wrong partition key.
⏱ Timestamps:
0:00 — The single log bottleneck
0:33 — Mental model: splitting into partitions
2:12 — Strategy 1: Round-robin routing (no key)
2:56 — Strategy 2: Key-based routing (murmur2 hashing)
4:03 — Strategy 3: Custom partitioner
4:25 — Partition count trade-offs
5:27 — The hot partition problem
6:20 — How to fix hot partitions
7:20 — Recap + what's next: Consumer Groups
We cover:
→ Why a single log hits a throughput ceiling
→ How partitions turn one log into a parallel, scalable system
→ Round-robin vs key-based vs custom routing
→ How murmur2 hashing determines partition assignment
→ The hot partition problem and how to fix it
→ Partition count trade-offs you need to know
This is Episode 2 in the Kafka Under The Hood series. Episode 1 covered what problem Kafka solves. This time we go one level deeper — into the scaling mechanism that makes Kafka handle millions of events per second.
📌 Series playlist: [ • Kafka Internals ]
📌 Episode 1 — What Problem Does Kafka Actually Solve?: [ • Kafka Internals #1 — What Problem Does Kaf... ]
Next episode: Producers, Consumers, and Consumer Groups — how readers coordinate across partitions.
---
Under The Hood Engineering
We explain distributed systems by showing you the internals — visually.
#kafka #systemdesign #distributedsystems #partitions #eventstreaming