How to Install & Run Apache Kafka: Docker, ZooKeeper & Python | Part 2
Learn how to install, configure, and run Apache Kafka locally! In Part 2 of our Kafka series, we move from theory to practice. I'll show you exactly how to set up Kafka using Docker Compose, how to do a manual installation with ZooKeeper, and how to write Python producers and consumers.
🚀 Want to see Part 3 about KRaft (ZooKeeper-less) production setup? Hit LIKE and comment "Part 3" down below!
👨💻 Get all the code and Docker files on GitHub:
https://github.com/GermanGerken/kafka...
📺 Missed Part 1? Watch the Kafka Architecture breakdown here:
• Apache Kafka Explained in 15 Minutes | Mes...
⏱️ TIMESTAMPS:
0:00 - Intro & What we're building
0:45 - Prerequisites (Docker & Python)
1:15 - Method 1: Running Kafka with Docker Compose
3:30 - Creating your first Kafka Topic via CLI
4:45 - Producing & Consuming messages (Command Line)
6:15 - Python Producer implementation (kafka-python)
8:00 - Python Consumer & Real-time streaming
9:30 - How Consumer Groups & Parallel Processing work
11:45 - Method 2: Manual Installation with ZooKeeper
13:30 - Kafka server.properties explained
15:15 - Where does Kafka actually store data on disk?
16:45 - Simulating a Multi-Broker Cluster locally
18:30 - Testing Fault Tolerance (Killing a Broker)
20:00 - Why ZooKeeper is dying (KRaft) & Part 3 Teaser
💡 In this tutorial, you will learn:
• How to write a docker-compose.yml file for Kafka and ZooKeeper
• How to use Kafka CLI tools (kafka-topics, console-producer, console-consumer)
• How to build real-time event streaming apps using Python
• How Consumer Groups automatically balance the load across partitions
• How to simulate a broker failure and watch Kafka's fault tolerance in action
• Where Kafka stores its binary log files on your hard drive
🛠️ Tech Stack Used:
Docker, Apache Kafka, ZooKeeper, Python 3, kafka-python library.
#ApacheKafka #Docker #Python #DataEngineering #MessageBroker #Backend #DistributedSystems #SoftwareEngineering #KafkaTutorial
🔔 Subscribe for more backend engineering, system design, and data architecture tutorials!