SFBigAnalytics_20200714: Productionalizing Big Data (talks from Lyft and Workday)
Talk #1: Can Apache Kafka Handle a Lyft Ride?
What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with.
In this talk, we'll be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We'll need to take care of your consumers as well to ensure that they don't process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters.
We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.
Speaker Bio:
Andrey Falko is a Staff Software Engineer at Lyft, where he has been for more than a year. He is currently focused on building and scaling reliable PubSub systems for Lyft's Data Platform. Prior to Lyft, Andrey worked at Salesforce for nine years where he researched Kafka and Pulsar performance and reliability. While there, he also built an IaaS system, many CI/CD systems, a Zipkin service, and features for the Salesforce platform.
Can Cecen is a Senior Software Engineer at Lyft, where he has been for a year. He works with the Streaming PubSub team focusing on the application layer and helping out teams across the company get their use-cases onboard to the pubsub infrastructure his team provides. Before Lyft, Can worked at Amazon Web Services for 7+ years, leading efforts to launch services such as Managed Streaming for Kafka and Lambda at Edge. He was part of the team that scaled CloudFront by 10x in a few years.
Talk #2: Optimize Cloud Services through Analyzing Operational Data on A Big Data Platform
To provide a high customer satisfaction, Workday leverages the operational data (both structured and unstructured data) to optimize services. At Workday, a centralized big data platform is used for descriptive, diagnostic and predictive analytics for all personas. This talk will introduce the challenges that the platform needs to address. Afterwards, we will present the tools used for various analysis tasks. Moreover, an in-house data pipeline platform will be presented, which is used for automation.
Speaker bio:
Lei Gao a senior machine learning engineer at Workday, who is leading a team to build a data science analytics platform.