Building Streaming on S3
Ryan Worl, co-founder and CTO of WarpStream, is on a mission to re-engineer fundamental infrastructure on top of S3.
We discuss the "aha" moment when discussing metrics at Uber with his now co-founder. How they took this insight and built a POC for their ideas before getting acquihired by DataDog. From there, they built Husky - DataDog's platform for events, logs and everything except metrics and then they thought:
"It will be cool to re-build Kafka on S3" and reached out to other developers to hear their thoughts. This lead them to build WarpStream - their current company.
In this episode we talk about Ryan's journey, how S3 can fundamentally change the way we design our infrastructure, and why it is a good idea to go in that direction. We talk about S3 vs S3 express. What will be the perfect serverless experience for streams. And finally, tune in at minute 50:00 to get excellent advice on how a SaaS provider should use logs.
--
00:00 Intros
00:28 Snowflake for Observability
04:15 Why Schema-less?
09:30 Benefits of S3-based architectures
15:15 Rethinking architectures for storage
20:29 The cost of running Kafka
23:10 What if we rewrite Kafka on S3?
28:45 Why not log to S3 directly from apps?
30:55 Multi-protocol streaming
34:00 Offering in-product analytics with S3 Express
38:50 Thinking in topics
42:15 Serverless Streams
45:33 Pay per use partitions
50:00 Advice: Use logs!