StreamNative Cloud: Deploying Elasticsearch Sink using pulsarctl
This video was created using StreamNative Hosted, a fully-hosted Pulsar cluster option available from StreamNative.
Quickly deploy the Pulsar Elasticsearch Sink Connector to StreamNative Cloud. This sink allows you to forward topic messages from StreamNative Cloud to Elasticsearch indices. In this video we deploy the Elasticsearch Connector using pulsarctl, send messages to the input topic using the Pulsar Java Client, and view those messages in Elasticsearch.
[00:00] Deploying Elasticsearch Sink Connector to StreamNativeCloud using pulsarctl
[00:28] Verify Elasticsearch version, obtain elasticSearchUrl, and create role and user
[02:54] Discuss StreamNative Cloud prerequisites prior to deploying Elasticsearch Sink Connector
[03:41] Connect to StreamNative Cloud cluster using pulsarctl
[05:46] Deploy Elasticsearch Sink Connector using puslarctl
[07:33] Verify connector has been deployed using StreamNative Cloud UI
[08:12] Test connector by producing messages to input topic using Pulsar Java Client
[09:15] Verify data has been sent to Elasticsearch by Elasticsearch Sink Connector
Prerequisites:
1. Create a service account and service account binding using snctl.
Detailed directions available here:
• Prepare Your Cluster Part 1: Create a Serv...
2. Create tenant and namespace, provide service account produce, consume and sinks permissions on the namespace using pulsarctl
**The ability to add sinks, sources, and packages permissions were added to the UI August, 2023. If you have this UI functionality you won't need to use pulsarctl in this step.**
Detailed directions available here (be sure to provide produce, consume, and sinks permissions):
• Prepare Your Cluster Part 2: Provide Servi...
If you are unfamiliar with using pulsarctl, the following video provides another example of downloading pulsarctl, getting pulsarctl connection information from StreamNative Cloud UI, and connecting using your service account (needed to deploy the sink as the service account):
• Deploying Pulsar Functions to SN Hosted wi...
pulsarctl command used in video to deploy sink connector
pulsarctl sinks create --archive builtin://elasticsearch \
--tenant elastictenant \
--namespace elasticnamespace \
--inputs elastictenant/elasticnamespace/elasticinput \
--name myelasticsink \
--subs-name myelasticsub \
--sink-config '{"elasticSearchUrl":"*****","indexName":"snindex","username":"snuser","password":"*****","schemaEnable":"true","createIndexIfNeeded":"true"}'
If you are unfamiliar with using Pulsar Java Client, the following video provides detailed directions on obtaining sample Java code from StreamNative UI and setting up Itellij:
• Test Pulsar Functions with SNProducer and ...
Elasticsearch Sink Connector documentation including list of fields to use in sink-config:
https://docs.streamnative.io/hub/conn...