Building a Self-Service Streaming Platform at Pinterest - Bairos Novak
Pinterest is a visual discovery engine that helps more than 250 million monthly active users discover things they love and inspires them to go do those things in their daily lives. Creating Pinterest's new stream processing platform, Xenon, around Flink has enabled teams across the company to tackle new real-time applications. From accelerating machine learning model training iterations to enabling real-time analytics, real-time stream processing has opened up new possibilities everywhere. Given that we ingest over a trillion messages every day through Kafka at Pinterest, a lot went into the design of our new stream processing platform.
We would like to share our experience in making the decision to move from different streaming technologies to Flink. We also discuss how we are building a self-service streaming platform for our engineers, data analysts and data scientists that provides a seamless job deployment pipeline, common monitoring, alerting and tooling. We have also enabled ad-hoc Flink SQL queries on bounded Kafka data streams through a UI to help users explore streaming data easily, extract real-time insights and assist product development.