Kafka Consumer Groups CLI Demo | Kafka-Python

Опубликовано: 28 Сентябрь 2024
на канале: Knowledge Amplifier

1,599

This video demonstrates the Kafka Consumer Groups concept using CLI

Prerequisite:
--------------------------
Consumer & Consumer Group in Kafka
   • Consumer & Consumer Group in Kafka

CLI Codes used
(this will only work if you installed kafka following the video:   • Install and run Apache Kafka & integr...  )

Start Zookeeper:
---------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/zookeeper-server-start.bat F:/kafka_2.12-3.2.0/config/zookeeper.properties

Start Kafka-server:
-----------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-server-start.bat F:/kafka_2.12-3.2.0/config/server.properties

Create topic:
------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-topics.bat --create --topic hello_world1 --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3

Without Consumer Group:
--------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic hello_world1 --from-beginning --bootstrap-server localhost:9092

F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic hello_world1 --from-beginning --bootstrap-server localhost:9092

Start Consumer with Group:
--------------------------------------------------------
F:/kafka_2.12-3.2.0/bin/windows/kafka-console-consumer.bat --topic hello_world1 --from-beginning --bootstrap-server localhost:9092 --group my-first-consumer-group

Producer Code:
--------------------------------
from time import sleep
from json import dumps
from kafka import KafkaProducer

topic_name='hello_world1'
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],value_serializer=lambda x: dumps(x).encode('utf-8'))

while True:
message=input("Enter the message you want to send : ")
partition_no=int(input("In which partition you want to send ? "))
producer.send(topic_name, value=message,partition=partition_no)

producer.close()

Check this playlist for more Data Engineering related videos:
   • Demystifying Data Engineering with Cl...

Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL