Iceberg Lakehouse Storage with Tabular and StreamNative BYOC Pulsar Clusters
This video demonstrates storing data from a StreamNative BYOC Pulsar cluster in an S3 bucket in Iceberg format using Tabular.
StreamNative requires only the warehouse name (training in example video) and Tabular credentials created in the Tabular profile to enable Lakehouse Storage with Iceberg/Tabular.
You must manually create a database in your Tabular warehouse with the name tenant_namespace corresponding to the Pulsar tenant and namespace.
Once StreamNative enables Lakehouse Storage at the cluster level, you must enable it on individual tenants/namespaces. This option is not yet available in the UI and can be completed using pulsar-admin. To use pulsar-admin from your machine, the endpoints must be public.
./bin/pulsar-admin \
--admin-url [admin url] \
--auth-plugin org.apache.pulsar.client.impl.auth.oauth2.AuthenticationOAuth2 \
--auth-params '{"privateKey":"file:///YOUR-KEY-FILE-PATH",
"issuerUrl":"[issuer url]",
"audience":"[audience]"}' \
namespaces set-offload-threshold [tenant]/[namespace] --size 0
At the time of recording this video, the first ledger of a topic must close before offloading to Lakehouse Storage will begin. After the first ledger rollover, the data can offload more frequently. The offload frequency shown in the video is every 2 minutes. If a topic is idle, it may require another ledger rollover to occur before offloading begins again. We encourage you to engage with StreamNative to determine how we can meet your specific Lakehouse Storage offloading needs.