Beyond Tiered Storage: Deep dive into WarpStream's Storage Engine (Richie Artoul) | Current 24
More: https://current.confluent.io | In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.
In this talk we'll cover how to design a storage engine from the ground up around commodity object storage. We'll dive deep into WarpStream's storage engine implementation and cover topics like: ingestion, metadata, compaction, fetch, and service discovery.
Finally, at the end we'll see how the concept of tiered storage can be re-introduced even in a system that has zero local disks to make different trade-offs around cost and performance.
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe on YouTube: / @confluentdeveloper
Community Slack: confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: / confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent