Building Data Lakehouse from Scratch - End to End Data Engineering Project
In this video you will learn to design, implement and maintain secure, scalable and cost effective lakehouse architectures leveraging Apache Spark, Apache Kafka, Apache Flink, Delta Lake, AWS, and open-source tools. Unlock data's full potential through advanced analytics and machine learning.
Part 2: • Realtime Streaming with Data Lakehouse - ...
FULL COURSE AVAILABLE: https://sh.datamasterylab.com/costsaver
Like this video?
Support us: / @codewithyu
Timestamps:
0:00 Introduction
1:24 The system architecture
4:59 The modern system architecture
9:15 Implementation of the Current Data Lakehouse on AWS Cloud
11:33 Creating Databases for Data Lakehouse
12:12 Using Glue crawler for Data Lakehouse
17:19 Using Lambda function to automate data orchestration on AWS Cloud
21:03 Coding the Lambda function
43:57 Optimising Lambda Function
48:46 Verification of Results
53:43 Outro
Resources:
Youtube Source Code:
https://buymeacoffee.com/yusuf.ganiyu...
🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
👦🏻 My Linkedin: / yusuf-ganiyu-b90140107
🚀 X(Twitter): https://x.com/YusufOGaniyu
📝 Medium: / yusuf.ganiyu
Hashtags:
#dataengineering #bigdata #dataanalytics #realtimeanalytics #streaming, #datalakehouse, #datalake, #datawarehouse, #dataintegration, #datatransformation, #datagovernance, #datasecurity, #apachespark, #apachekafka, #apacheflink, #deltalake, #aws, #opensource, #dataingestion, #structureddata, #unstructureddata, #semi-structureddata, #dataanalysis, #advancedanalytics, #dataarchitecture, #costoptimization, #cloudcomputing, #awscloud