Apache Kafka Log Compaction in Action: Deduplication Demo
A quick technical demonstration showing how Apache Kafka handles message deduplication at the broker level using the compact cleanup policy.
In this demo, you’ll see multiple messages produced with the same key and how the Log Cleaner thread processes the segments to retain only the most recent value for each unique key. This illustrates how Kafka maintains a "latest state" snapshot of your data while reducing storage overhead.
🚀 What’s Shown:
The Ingest: Streaming multiple updates to the same record keys.
The Result: The state of the "clean" log after compaction, showing successful deduplication.
⚙️ Key Configuration:
Property: cleanup.policy
Value: compact
Goal: Efficient state management and automated deduplication of keyed data.
⚙️ Applied Configuration
To trigger the compaction seen in this video, the following configuration was applied:
bin/kafka-configs --bootstrap-server localhost:9092 \
--entity-type topics \
--entity-name [TOPIC_NAME] \
--alter \
--add-config 'cleanup.policy=[compact,delete],max.compaction.lag.ms=120000,min.cleanable.dirty.ratio=0.01'
#Kafka #ApacheKafka #DataEngineering #Shorts #TechDemo #EventStreaming