What makes Apache Pinot so Fast?
For memberships: join this channel as a member here:
/ @thegeeknarrator
Summary:
In this episode, host Kaivalya Apte interviews Ankit Sultana, a staff engineer at Uber with extensive experience in Apache Pinot, a real-time analytics platform. They discuss the high-level architecture, ingestion processes, and query mechanisms of Apache Pinot. Ankit provides a historical context, detailing the evolution of Apache Pinot from its origins at LinkedIn to its widespread adoption. They discuss the key components of Pinot, explaining the roles of Pinot servers, brokers, controllers, and the dependency on Zookeeper. Ankit also explained how data flows into Apache Pinot and the technicalities of its real-time ingestion and querying capabilities.
Chapters:
00:00 Introduction and Episode Overview
03:30 Understanding Apache Pinot
03:49 Apache Pinot's Historical Background
05:20 Real-Time Analytics with Apache Pinot
11:06 Apache Pinot's Architecture and Components
17:05 Tenancy and Data Ingestion in Apache Pinot
30:22 Understanding Real-Time Replication and Consumer Groups
30:52 Pinot's Offset Tracking and Segment Creation
31:59 Handling Server Restarts and Segment Transitions
32:50 Dealing with Kafka Duplicates and Deduplication Features
35:13 Ingestion Process and Mutable vs Immutable Segments
39:18 Memory Management and Segment Flushing
40:10 Advantages of Keeping Mutable Segments Longer
42:21 Introduction to Pinot's Query Engines
42:50 Single Stage Engine: Architecture and Optimizations
54:49 Multi-Stage Engine: Flexibility and Challenges
58:13 Conclusion and Next Steps
Important Links:
Good high-level overview on Pinot: • Meetup: Pinot, Why Are You So Fast? (Chinm...
Apache Pinot 101 by Tim: • Apache Pinot 101
Multistage Physical Optimizer, the new optimizer that we built at Uber and open-sourced: https://docs.pinot.apache.org/users/u...
Multistage Lite Mode: https://docs.pinot.apache.org/users/u...
Time Series Engine Talk at RTA Summit: • Time Series Query Engine for Apache Pinot ...
For memberships: join this channel as a member here:
/ @thegeeknarrator
Don't forget to like, share, and subscribe for more insights!
=============================================================================
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
https://app.codecrafters.io/join?via=...
=============================================================================
Database internals series: • Write-ahead-logging
Popular playlists:
Realtime streaming systems: • Realtime Streaming Systems
Software Engineering: • Software Engineering
Distributed systems and databases: • Distributed Systems and Databases
Modern databases: • Modern Databases
Stay Curios! Keep Learning!