How to build ETL pipeline with Incremental Data Load with Python | Python | ETL
In this video we will cover how to perform Incremental data load with Python ETL pipeline. The incremental data load approach in ETL (Extract, Transform and Load) is the ideal design pattern. In this process, we identify and process new and modified rows since the last ETL run. Incremental data load is efficient in the sense that we only process a subset of rows and it utilizes less resources.
We will focus on Destination Change Comparison technique today.
Join me next time for the Source Change Detection.
Link to medium article on this topic:: https://blog.devgenius.io/python-etl-...
Change Data Capture approach video: • SSIS Incremental load with SQL Server...
Link to GitHub repo:: https://github.com/hnawaz007/pythonda...
Subscribe to our channel:
/ haqnawaz
---------------------------------------------
Follow me on social media!
Github: https://github.com/hnawaz007
Instagram: / bi_insights_inc
LinkedIn: / haq-nawaz
---------------------------------------------
#ETL #Python #IncrementalDataLoad
Topics covered in this video:
0:00 - Introduction to ETL Incremental load approach
1:27 - Different ETL Incremental load approaches
2:46 - Implement Incremental load approach with Python
5:17 - Test Pipeline with target data check