Integration of AWS Glue Schema Registry & Kafka Consumer using Python
This video explains the integration of AWS Glue Schema Registry & Kafka Consumer using Python.
Prerequisites:
---------------------------
Introduction to Schema Registry in Kafka | Part 1
• Introduction to Schema Registry in Ka...
Introduction to Schema Registry in Kafka | Part 2
• Introduction to Schema Registry in Ka...
Using Glue schema registry for Apache Kafka with Python
• Using Glue schema registry for Apache...
Consumer Code:
------------------
#pip3 install boto3 -t.
#pip3 install aws-glue-schema-registry --upgrade --use-pep517 -t .
#pip install kafka-python -t .
import boto3
from aws_schema_registry import SchemaRegistryClient
from kafka import TopicPartition , OffsetAndMetadata
from aws_schema_registry.adapter.kafka import KafkaDeserializer
from kafka import KafkaConsumer
session = boto3.Session(aws_access_key_id='{}', aws_secret_access_key='{}')
glue_client = session.client('glue', region_name='us-east-1')
Create the schema registry client, which is a façade around the boto3 glue client
client = SchemaRegistryClient(glue_client,
registry_name='my-registry')
Create the deserializer
deserializer = KafkaDeserializer(client)
consumer = KafkaConsumer ('topic2',bootstrap_servers = ['{}'],
value_deserializer=deserializer,group_id='{}',auto_offset_reset='earliest',
enable_auto_commit =False)
for message in consumer:
print(message)
print("The value is : {}".format(message.value))
print("The key is : {}".format(message.key))
print("The topic is : {}".format(message.topic))
print("The partition is : {}".format(message.partition))
print("The offset is : {}".format(message.offset))
print("The timestamp is : {}".format(message.timestamp))
tp=TopicPartition(message.topic,message.partition)
om = OffsetAndMetadata(message.offset+1, message.timestamp)
consumer.commit({tp:om})
print('*' * 100)
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engineering with Cl...
Apache Kafka form scratch
• Apache Kafka for Python Developers
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL