Azure Databricks/EventHub - Build a Delta Lake with PySpark Streaming from scratch - End to End
In this video we build a Delta Lake using PySpark Streaming from scratch. We stream the data received by EventHub to Databricks and utilizing PySpark we create three layers in our Storage Account according to Medallion architecture. The example is based on Databricks documentation tutorial:https://www.databricks.com/notebooks/...
Documentation for PySpark and Databricks used: https://docs.databricks.com/structure... and https://docs.databricks.com/delta/tun...
The code can be found at: https://github.com/apostolos1927/Stre...
Follow me on social media:
LinkedIn: / apostolos-athanasiou-9a0baa119
GitHub: https://github.com/apostolos1927/
Facebook: https://web.facebook.com/profile.php?...
00:00 - Intro
01:03 - Architecture of the Delta Lake
02:12 - Building the Delta Lake step by step using PySpark Streaming
21:57 - Conclusion