Azure Databricks Dynamic Notebook Trigger and Transformation from Azure Data Factory Pipeline
Learn how to invoke Databricks notebook from Azure data factory and pass dynamic content to the notebook as JSON payload from Azure SQL server to perform dynamic PySpark transformations.
In this hands-on demo, you will learn how to query SQL server from Azure data factory to parse selective configurations and pass them to Azure Databricks activity as input. Parallely, you will learn how to create a Databricks linked service for ADF and read filtered data from the Azure SQL server using a JDBC spark connection.
Inbuilt dynamic data transformations with robust pipeline structuring and exploring how to use different ADF activities are a part of the demo.
00:00 - Overview
00:45 - Lookup and Set Variable Activity
05:25 - Databricks Linked Service Creation
08:05 - Databricks Notebook Activity Setup
08:28 - Databricks Notebook walkthrough
12:05 - Pipeline Debug
LET'S CONNECT!
📰 LinkedIn ➔ / jayachandra-sekhar-reddy
🐦 Twitter ➔ / reddyjaysekhar
📖Medium ➔ / jay-reddy
📲 Substack➔ https://databracket.substack.com
💁Fiverr ➔ https://www.fiverr.com/jayreddy9
#pyspark #databricks #azuredatafactory #azuredataengineer #azuredatabricks #dataengineering #dataengineers #pythonprogramming #bigdata #bigdatatutorial