Kafka streams at scale
Problem Statement: Stateful Realtime Processing of multi-million events.
Intro Kafka Streams and event flow (2 slides)
Challenges in Kafka Streams
a. Fault Recovery
b. Horizontal Scalability
c. Cloud Readiness
d. Restricted RocksDB
e. Large Clusters
Lay a background on why are these a challenge.
How we forked the code to solve each of these over the past year.
Conclusion
Future Works
Deepak is working at Walmart Labs as a software engineer in the Customer Backbone team where multi-millions events need to be processed in real-time. He’s promoting the event-driven architecture, enabled by a “Distributed NoSQL DB and Streaming Platform” based on Kafka Streams that the team is working on. He’s now working on a Distributed Graph Algorithm using the event-driven architecture.