Kora: Cloud Native Platform for Kafka
We invited Anna Povzner, director of engineering at Confluent, to the show to discuss Kora. Kora is a Cloud Native platform based on Apache Kafka, which her team at Confluent built. Anna and her team recently published a paper about Kora in VLDB 2023, and it won the industry's best paper award.
Kora VLDB paper: https://www.vldb.org/pvldb/vol16/p382...
In our conversation, we discussed:
Capacity abstractions for serverless Kafka
Reconciling design constraints into a cohesive architecture
Key components of the architecture: Rebalances, quotas, cells.
How to drive deep technical innovation at a fast-growing company
Delivering low and predictable latency with Apache Kafka
---
00:00 Intro
2:00 Abstractions are key to Serverless
6:25 Unifying Dedicated and Multi-Tenant Kafka
7:35 Reconciling design constraints
11:50 Architecture for unexpected workloads
19:16 Dynamic utilization management
21:30 Benefits Cell architecture
28:00 Driving innovation in a growth company
33:30 Performance requirements
38:00 Advice for building SaaS on Kafka