How to perform ETL Incremental Data Load using DLT | Data Load Tool | ETL | Python
In this video we will continue with the data load tool (dlt) library. We will explore how to perform incremental data load using dlt. The incremental data load in ETL (Extract, Transform and Load) is the act of loading only new or changed data. With this approach we process minimal data, use fewer resources and therefore less time. DLT refers to this as the merge write disposition.
We keep the lastest snapshot of a recrod in the data warehouse. We update and/or insert a new record in the dimension table. This is referred to as upsert.
Link to GitHub repo: https://github.com/hnawaz007/pythonda...
Link to previous video (db setup): • data load tool (dlt) build database d...
Python based Incremental laod (Source Change Detection): • Python ETL pipeline how to perform in...
Python based incremental load (Destination Change Comparison) : • How to build ETL pipeline with Increm...
DLT docs on incremental load: https://dlthub.com/docs/general-usage...
Link to Channel's site:
https://hnawaz007.github.io/
--------------------------------------------------------------
💥Subscribe to our channel:
/ haqnawaz
📌 Links
-----------------------------------------
Follow me on social media!
🔗 GitHub: https://github.com/hnawaz007
📸 Instagram: / bi_insights_inc
📝 LinkedIn: / haq-nawaz
🔗 / hnawaz100
🚀 https://hnawaz007.github.io/
-----------------------------------------
#ETL #incremental #dlt
Topics in this video (click to jump around):
==================================
0:00 - Introduction to data load tool (dlt) incremental load
0:42 - Source Change Detection: Merge Write Disposition
1:38 - How Merge Write Disposition works
2:08 - Source SQL Server DB setup
2:33 - DLT Incremental Load Function
3:27 - Test Incremental Load Function
5:03 - Update/Insert records in Source SQL DB
5:29 - Run the dlt pipeline
5:36 - Review pipelines results
6:00 - Coming Soon