kdb+ and Kafka - Multi Broker using KRaft Mode
In this video I give some background theory on how Kafka works, demonstrate how Kafka Brokers can be started, with and without Zookeeper, and then show how to create kdb+ based producers and consumers
Support the channel - donation addresses:
Ethereum (ERC-20) 0xC71782F6e3CF95eb6d113816023A6141e411fA43
Bitcoin bc1qcg70h076jw3rwyammxj5qs2wldktezp9nvgfke
Online Resources:
Apache Krafka download: https://kafka.apache.org/documentatio...
librdkafka-dev: https://github.com/edenhill/librdkafka
librdkafka-dev config: https://github.com/edenhill/librdkafk...
Code.kx Kafka page: https://code.kx.com/q/interfaces/kafka/
kdb+kfk Library: https://github.com/KxSystems/kafka
kfk Functions reference: https://code.kx.com/q/interfaces/kafk...
AquaQ Kafka doc: https://aquaq.co.uk/kafka-kdb-java/
Chapters:
0:00 Introduction
00:19 Why is Kafka worth learning
01:02 Kafka in bullets
02:08 Architectural Diagram
04:27 Kafka Partitions explained
06:59 Kafka Replication explained
08:34 Kafka Producers
09:30 Kafka Acknowledgements
10:14 Kafka Consumers
11:58 Online Resources
14:38 Installing librdkafka-dev
16:05 Installing kdb+ kfk kafka library
18:18 Installing Kafka binaries
19:38 Check Kafka version
19:55 Starting and Stopping Zookeeper
22:30 Starting a single Kafka Broker
24:04 What is KRaft Mode
24:45 Configure KRaft, create unique ID
25:40 KRaft sample configs
28:07 Modify config files to create 3 nodes
31:44 Using Cluster ID to modify storage directory
33:02 Starting 3 Kafka Brokers in KRaft Mode
33:59 Manual Topic creation
36:10 Dynamic Topic creation
39:02 kdb+ Creating a Producer
47:35 kdb+ Enabling Producer acknowledgements
50:25 Where are Kafka records stored on the server
52:05 Decoding Kafka .log file command
53:37 kdb+ Producer Check the statistics table
55:14 kdb+ Creating a Consumer
59:50 1 Consumer, multiple partitions test
1:00:50 Decode json messages, .j.k
1:02:20 Decode serialise/deseralise -8!/-9!
1:03:05 Consumer read from beginning of Partition
1:05:37 2 Consumers, multiple partition test
1:11:08 2 Consumers, different consumer group
1:13:12 kdb+Topic Producer config
1:14:25 kdb+ Producer, using keys
1:15:55 kdb+ Benefit of keys
1:16:49 More online resources and closing
#kdb, #kafka