COPY ACTIVITY CONTIANER TO CONTIANER DATALAKE TO AZURE SQL PIPELINE META DATA
To create a pipeline with a Copy Activity that copies data from a container in Data Lake Storage to a container in Azure SQL Database using metadata-driven execution in Azure Data Factory, you can follow these steps:
1. Sign in to the Azure portal (portal.azure.com) using your Azure account.
2. In the Azure portal, search for and open "Azure Data Factory."
3. In the Azure Data Factory page, click on your Data Factory instance or create a new one if you haven't done so already.
4. In your Data Factory instance, navigate to the "Author & Monitor" section.
Creating Linked Services for Data Lake Storage and Azure SQL Database:
1. Click on the "Author" button to open the Data Factory authoring environment.
2. In the authoring environment, click on the "Manage" tab on the left side and select "Linked services."
3. Click on the "+ New" button to create a new linked service.
4. Choose the appropriate linked service types for Data Lake Storage and Azure SQL Database.
5. Provide a unique name for each linked service.
6. Configure the connection settings for each linked service, including the necessary credentials and connection details.
7. Click on the "Test connection" button to validate the connections to Data Lake Storage and Azure SQL Database.
8. Once the connections are successfully tested, click on the "Create" button to create the linked services.
Creating a Pipeline with Metadata-Driven Execution:
1. In the authoring environment, click on the "Author & Monitor" tab on the left side and select "Author."
2. Click on the "+ New pipeline" button to create a new pipeline.
3. Give your pipeline a name and optionally provide a description.
4. Drag and drop the "Copy Data" activity from the "Activities" pane on the left side onto the canvas.
5. Configure the "Copy Data" activity by providing the necessary details:
a. Set the source dataset to the Data Lake Storage dataset pointing to the container you want to copy from.
b. Set the sink dataset to the Azure SQL Database dataset pointing to the container where you want to copy the data.
6. In the "Source" tab of the "Copy Data" activity, select the "Metadata-driven" execution property.
7. Configure the metadata-driven execution properties, such as the metadata file location, metadata file path column, and additional settings based on your requirements.
8. Click on the "Mappings" tab to define any transformations or mappings required during the data copy process.
9. Once you have defined the "Copy Data" activity and its settings, click on the "Publish all" button to publish your pipeline.
10. After publishing, you can trigger the pipeline manually or schedule it to run at specific intervals.
By following these steps, you will have created a pipeline with a Copy Activity that copies data from a container in Data Lake Storage to a container in Azure SQL Database using metadata-driven execution in Azure Data Factory. Make sure to configure the appropriate access permissions and credentials for the Data Lake Storage and Azure SQL Database resources to ensure successful data transfer.