SQL On The Edge Ep 20 - Azure Data Factory Mapping Data Flows
Azure Data Factory (ADF) Mapping Data Flows (MDFs) are a new way to do data transformation activities inside ADF without the use of code. Before MDFs, ADF did not really have transformation capabilities inside the service, it was more ELT than ETL. The idea was to use ADF to move data around and then use the compute power of the data sinks to perform transformation.
With MDFs, now we have a way to do real ETL, with the transformation defined and executed from inside ADF. Behind the scenes, ADF uses the Azure Databricks service to spin up and down clusters as needed to perform the transformations. These data flows can be integrated as one activity of a bigger pipeline, expanding the possibilities of what can be built and managed from inside ADF without dealing with more external services.
In this video we will go over the value proposition of Mapping Data Flows and show an example of a data flow authoring and monitoring experience.