Azure EventHub & Databricks - Delta Tables with Streaming Data using Python and PySpark
In this video we create Delta tables with the data we receive on Azure EventHub. We generate random json data using Python and send it to Azure EventHub. From there, utilizing Spark Streaming in Databricks we receive the incoming data in real time and save it into Delta tables. The code for this example can be found on GitHub: https://github.com/apostolos1927/Data...
The Maven library used is: com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22
Follow me on social media:
LinkedIn: / apostolos-athanasiou-9a0baa119
GitHub: https://github.com/apostolos1927/
Facebook: https://www.facebook.com/profile.php?...
00:00 - Intro
00:59 - Sending data to Azure EventHub with Python
08:36 - Setting up Azure Databricks
11:53 - Spark Code for Streaming Data
23:23 - Spark Code Second Approach
25:26 - Conclusion