Real-time Data Infrastructure - At Uber and Beyond
Pinot team at Uber wrote a great paper about the real-time analytics platform that they built. Chinmay, formerly a principal engineer at Uber and now head of product at StarTree, joined me for a conversation.
We discussed the challenges they encountered at Uber, the solutions they came up with, the platform they built and how to best apply their experience to companies much smaller than Uber.
The paper: https://arxiv.org/pdf/2104.00087.pdf
00:00 Intro
01:08 Fundamental challenges for real-time data at Uber
07:21 Getting started with real-time
10:00 Dealing with competing requirements
18:44 Growth of real-time at Uber
19:49 Real-time data stack
20:45 Mapping technology to use-case
24:27 How 4 person team built all this?
25:21 Presto query federation
27:55 Example of mapping tech to a use-case
29:30 Making use-case evolution transparent
33:30 Leading product at Startree