Kafka on StreamNative (KSN): Topic Compaction
This video was created with Kafka on StreamNative 3.1 using a StreamNative Hosted Pulsar Cluster. Kafka on StreamNative brings Enterprise-grade Kafka support with support SLA to StreamNative Pulsar Clusters. StreamNative Hosted is a fully-hosted Pulsar Cluster option available from StreamNative.
Using topic compaction with Kafka consumers.
[00:00] Topic compaction
[00:11] Use kafka-topic.sh script to create a new topic with cleanup-policy=compact
[01:30] Discuss frequency of topic compaction and compaction threshold (default 100MB), set compaction threshold to 1MB using pulsarctl
[01:58] Test topic compaction by producing at least 1MB of keyed messages, allow topic compaction to complete, consume messages using Kafka consumer to view messages in compacted topic
View the video on querying consumer backlog for download and setup instructions to use Kafka scripts in the bin folder.
The following command can be used to create a new topic in public/default with topic compaction enabled:
./bin/kafka-topics.sh --bootstrap-server "{server url}:9093" --command-config ~/kafka/kafka-properties --create --topic {topic name without tenant or namespace} --partitions 1 --config cleanup.policy=compact
pulsarctl can be used to edit an existing topic in any tenant/namespace to enable topic compaction. This will cause the Kafka consumer to use the compacted topic. Unlike with a Pulsar consumer, if topic compaction is enabled, the Kafka consumer will always use the compacted topic.
Directions on downloading and configuring security for puslarctl can be found in this video:
• Deploying Pulsar Functions to SN Hosted wi...
The following pulsarctl command can then be used to enable topic compaction and set the topic compaction threshold for an existing topic in any tenant/namespace:
pulsarctl topics set-compaction-threshold -t 1M public/default/compactedtopic
It's also possible to enable topic compaction at the namespace level. This will enable topic compaction for all topic in that namespace.