Building a new Database Query Optimiser - @cmu
Read more about Kafka Diskless-topics, KIP by Aiven:
KIP-1150: https://fnf.dev/3EuL7mv
Summary:
In this conversation, Kaivalya Apte and Alexis Schlomer discuss the internals of query optimization with the new project optd. They explore the challenges faced by existing query optimizers, the importance of cost models, and the advantages of using Rust for performance and safety. The discussion also covers the innovative streaming model of query execution, feedback mechanisms for refining optimizations, and the future developments planned for optd, including support for various databases and enhanced cost models.
Chapters
00:00 Introduction to optd and Its Purpose
03:57 Understanding Query Optimization and Its Importance
10:26 Defining Query Optimization and Its Challenges
17:32 Exploring the Limitations of Existing Optimizers
21:39 The Role of Calcite in Query Optimization
26:54 The Need for a Domain-Specific Language
40:10 Advantages of Using Rust for optd
44:37 High-Level Overview of optd's Functionality
48:36 Optimizing Query Execution with Coroutines
50:03 Streaming Model for Query Optimization
51:36 Client Interaction and Feedback Mechanism
54:18 Adaptive Decision Making in Query Execution
54:56 Persistent Memoization for Enhanced Performance
57:12 Guided Scheduling in Query Optimization
59:55 Balancing Execution Time and Optimization
01:01:43 Understanding Cost Models in Query Optimization
01:04:22 Exploring Storage Solutions for Query Optimization
01:07:13 Enhancing Observability and Caching Mechanisms
01:07:44 Future Optimizations and System Improvements
01:18:02 Challenges in Query Optimization Development
01:20:33 Upcoming Features and Roadmap for optd
References:
NeuroCard: learned Cardinality Estimation: https://vldb.org/pvldb/vol14/p61-yang...
RL-based QO: https://arxiv.org/pdf/1808.03196
Microsoft book about QO: https://www.microsoft.com/en-us/resea...
Cascades paper: https://15721.courses.cs.cmu.edu/spri...
optd source code: https://github.com/cmu-db/optd
optd website (for now): https://db.cs.cmu.edu/projects/optd/
For memberships: join this channel as a member here:
/ @thegeeknarrator
Don't forget to like, share, and subscribe for more insights!
=============================================================================
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
https://app.codecrafters.io/join?via=...
=============================================================================
Database internals series: • Write-ahead-logging
Popular playlists:
Realtime streaming systems: • Realtime Streaming Systems
Software Engineering: • Software Engineering
Distributed systems and databases: • Distributed Systems and Databases
Modern databases: • Modern Databases
Stay Curios! Keep Learning!
#database #queryoptimization #sql #postgres