Scaling Derived Data for Planet-Scale Applications at Linkedin
In this video I speak with Felix GV, who is a Principal Staff Engineer at Linkedin, and has done major contributions to the data infrastructure and Linkedin, including VeniceDB.
This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads.
Chapters:
00:00 Introduction
01:42 The Evolution of LinkedIn's Databases
03:15 Challenges with Voldemort and the Birth of VeniceDB
08:42 Understanding Derived Data
13:33 Planet-Scale Applications and Multi-Region Support
17:40 Writing Data into VeniceDB
22:53 Merging Data in VeniceDB
40:31 Understanding the Architecture
40:47 Components of the Write Path
41:56 Leader and Follower Architecture
43:58 Partitioning and DaVinci Client
47:57 Read Patterns and Client Options
54:25 Fault Tolerance and Recommender Systems
01:01:19 Kafka Integration and Deployment
01:06:56 Roadmap and Future Improvements
Important links:
VeniceDB blog: / open-sourcing-venice-linkedin-s-derived-da...
VeniceDB docs: https://venicedb.org/
Qcon: • "Open-Sourcing Venice" by Felix GV (S...
Follow me on Linkedin and Twitter: / kaivalyaapte and / thegeeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: • Write-ahead-logging
Popular playlists:
Realtime streaming systems: • Realtime Streaming Systems
Software Engineering: • Software Engineering
Distributed systems and databases: • Distributed Systems and Databases
Modern databases: • Modern Databases
Stay Curios! Keep Learning!
#kafka #linkedin #venicedb #Rocksdb