Kafka on StreamNative (KSN): Message Retention and Kafka Consumer Groups
This video was created with Kafka on StreamNative 3.1 using a StreamNative Hosted Pulsar Cluster. Kafka on StreamNative brings Enterprise-grade Kafka support with support SLA to StreamNative Pulsar Clusters. StreamNative Hosted is a fully-hosted Pulsar Cluster option available from StreamNative.
Enable message retention for Pulsar topics when using Kafka consumer groups. Skip to 04:33 to enable message retention using StreamNative UI.
[00:00] Message retention and Kafka consumer groups
[00:11] View entries in segments/ledgers of our Pulsar topic
[01:16] Discussion of when ledgers can rollover and be eligible for deletion
[02:11] Produce 50k messages to force a ledger rollover, show that the ledger is deleted because no retention policy is set, consume remaining messages
[04:33] Enable message retention in UI
[05:25] Produce 50k messages after message retention is enabled to show the ledger is not deleted
Note that message retention works differently for Pulsar consumers compared to a Kafka consumer group. In the case of a Pulsar consumer, messages inside a subscription that have not yet been consumed by default will not be deleted even if message retention is not set.