Kafka Topics, Partitions & Consumer Groups | Explained with Real Examples
A deep dive into topic partitions, and its relationship with consumers and consumer-groups.
#apachekafka #dataengineering #datastreaming
Code: https://github.com/maxcotec/apache-kafka
Part 1 - • Apache Kafka for beginners | A practical g...
Video outline
00:00 Intro and video outline
01:47 subscribe!
02:17 quick recap
02:37 what is a kafka Topic?
03:03 how topic partitions helps
05:14 topic partition offsets
06:20 real example - 10k/sec events
07:02 problem with single partition
07:16 solve using multi-partition
09:40 consumer groups
12:28 Vertical scaling limitation
14:45 Horizontal scaling
15:50 Consumer-group offsets
17:58 Multiple consumer-groups
18:55 How many partitions can we have?