Pinterest’s Journey to a Automated, Efficient & Low-Maint. PubSub Platform (Pinterest) | Current 24
More: https://current.confluent.io | At Pinterest, Apache® Kafka runs on ~2000 brokers across 50+ clusters, moving data at petabyte scale every day. Through investing in automation and focusing on cost efficiency, we have seen positive results in dev velocity, scalability, stability, and cost footprint, enabling everyone to quickly and easily launch data streaming use cases that power business growth, while keeping our platform low-maintenance.
We believe that sharing the innovations along our journey to a highly-automated, efficient, and low-maintenance PubSub platform can provide immense value for industry partners. This talk will cover the following topics in-depth, bringing inspiration and actionable takeaways for each attendee who might face the same problems operating PubSub platforms at scale:
1. Automation: 80% reduction in overall onboarding time for PubSub applications
Automated endpoint discovery with PubSub Client (PSC) via 100% completion of Java migrations to PSC, available standalone and as Flink connector (open-sourced)
Automated resource provisioning for PubSub-based applications via drag-and-drop UI, reducing setup time from 30 mins to 5 mins
Automated Kafka broker & cluster management and recovery via Orion (open-sourced)
Automated client-side error handling via PSC (open-sourced)
2. Efficiency: 25% reduction in platform cost
Reducing PubSub infrastructure cost via migrating large pipelines from Kafka to MemQ (open-sourced)
Reducing Kafka cluster sizes and optimizing efficiency via using heterogeneous hardware
Reducing Kafka broker footprint via storage tiering
Reducing cross-region KafkaMirror resource utilization via Shallow Mirror (KIP-712)
3. Low-Maintenance: simplifying business processes
Generic CDC framework via Debezium / Kafka Connect
Improving cost visibility and attribution for multi-tenant environment, enabling cost-driven decision making
Moving towards service tiering to further simplify customer onboarding and platform operation.
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe on YouTube: / @confluentdeveloper
Community Slack: confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: / confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent