How to Build an ETL Analytics Pipeline With Databricks Using Airflow!
In this video, I'll be going through how you can run and monitor Databricks workflows using Airflow so you can have a full picture of every step of your data pipeline in one place! This is a great tool if you have steps in your data pipeline before or after your Databricks notebooks, so that you can monitor it all from a single UI, rather than constantly switching between management UI's. Check out the code and follow along at the link below:
https://docs.astronomer.io/learn/airf...
🎬 Watch My Daily Data & AI Shorts: / @dataandaiguyshortform