Complete End to End Data Engineering Project In Hindi | Pyspark | Databricks | Data Factory | SQL
Resources & Links Mentioned:
raw data Github Source Dataset Link : https://raw.githubusercontent.com/Abh...
Dataset : https://arbrecreations-my.sharepoint....
Timestamp:
[00:00:00] Introduction and Objective of Data Engineering
[00:01:07] The Single Purpose of Data Engineering
[00:02:54] Limitations of Excel-Based Processes
[00:04:13] Introduction to ETL and Automation
[00:07:25] Limitations of SQL; OLTP vs OLAP
[00:12:54] What is a Data Warehouse and Why It Is Needed
[00:15:14] Data Scientist vs Data Engineer; Data Lake and ELT
[00:20:10] Medallion Architecture Overview
[00:20:53] Layering Concept in Lakehouse
[00:22:04] Batch vs Streaming Differences
[00:22:49] Bronze / Silver / Gold Layer Definitions
[00:26:06] Benefits: Data Quality and Fault Isolation
[00:27:34] First Pipeline: GitHub to Azure SQL
[00:27:59] Project Overview and Schema
[00:32:48] Azure Account and Resource Group Setup
[00:39:08] GitHub Account, Repository, and CSV Upload
[00:46:10] Azure SQL Database and Table Creation
[00:58:00] Creating ADF Factory and Copy Activity
[01:04:45] HTTP Linked Service and Parameterized Dataset
[01:16:00] SQL Linked Service and Table Mapping
[01:19:47] Debug Run and Data Verification
[01:21:58] Publish and Wrap-Up
[01:23:16] Incremental Load and ADLS Bronze Layer
[01:23:22] Incremental Load and Watermark Table Concept
[01:26:15] ADLS Account and Containers (Bronze / Silver / Gold)
[01:37:21] Lookup Activity: Reading Last/Max Date
[01:45:20] Dynamic Date Filter in Source Query
[01:49:24] Sink to Bronze Parquet and Linked Service
[01:51:58] Stored Procedure Creation and Parameters
[01:59:08] File and Watermark Table Validation After Run
[02:02:40] Loading Bronze Parquet into Databricks
[02:03:07] Workspace and Cluster Setup
[02:07:44] Notebook Connection and Imports
[02:10:49] ADLS Connection Configuration and Access Keys
[02:19:25] File Listing, Naming, and Parquet Read
[02:23:33] Silver Layer: Cleaning and Writing Data
[02:24:48] Schema Validation
[02:26:12] Converting Text Columns to Title Case
[02:29:04] Removing Missing and Duplicate Rows
[02:32:33] Overwriting Data into Silver Parquet
[02:34:52] Silver Data Validation in ADLS
[02:36:00] Gold Layer Introduction and Customer Dimension (Basics)
[02:37:31] Understanding Dimension vs Fact
[02:52:05] Distinct Customer Query
[02:53:55] Creating Source DataFrame
[02:55:22] Source and Sink Concept
[02:57:00] SCD Type-1 Upsert and Surrogate Key (Customer)
[02:57:12] Understanding Upsert (Update + Insert)
[03:03:17] Importance of Surrogate Keys
[03:07:15] Separating Old and New Records Using Left Join
[03:09:26] Assigning Keys to New Records and Union
[03:12:11] Delta Table Upsert and Catalog Fix
[03:49:00] First-Time Delta Write (save)
[03:56:00] Delta Merge and Verification
[03:52:55] Catalog vs Delta: Fixing Exists Check
[03:59:00] Applying Changes Across All Dimension Scripts
[04:00:14] Gold Layer: Product Dimension
[04:02:02] Cloning Customer Notebook for Product
[04:03:20] Selecting Product Columns
[04:07:11] Join, Key Assignment, and Upsert Execution
[04:10:53] Table Verification
[04:11:48] Gold Layer: Date Dimension
[04:14:28] Distinct Order Date and Schema
[04:16:45] Extracting Year, Month, and Day Columns
[04:20:21] Date Dimension Schema and Join
[04:24:59] Upsert and Validation
[04:25:17] Gold Layer: Region Dimension
[04:27:15] Distinct Region Source
[04:28:41] Region Dimension Schema and Join
[04:31:10] Upsert and Table Check
[04:31:51] Fact Table Creation
[04:35:26] Loading All Dimensions
[04:41:02] Loading Silver Data
[04:42:54] Four Left Joins and Column Selection
[04:48:37] Why Fact Tables Contain Only Keys
[04:51:03] Removing Catalog Dependency and Applying Delta Check
[05:05:24] Gold Files Verification
[05:11:46] Databricks Jobs Pipeline
[05:12:44] Dependency Design (Job Graph)
[05:15:05] Silver Task Setup
[05:16:52] Gold Customer / Product / Date / Region Tasks
[05:23:58] Fact Sales with Multiple Dependencies
[05:25:26] Run and Fix: Filter vs Aggregate Error
[05:35:25] Cleanup and Successful Re-Run
[05:39:56] ADF Pipeline with Databricks Integration
[05:40:53] Databricks Linked Service and Token
[05:42:48] Base Parameters and Triggers
[05:43:24] Project Recap
What you will learn in this project:
Designing an end-to-end data engineering pipeline
Data ingestion using (ADF)
Implementing Medallion Architecture
Data transformation using Spark and Spark SQL
Writing SQL queries
Implementing upsert and merge logic
Handling incremental data loads
Building pipelines in Databricks
Real-world best practices
If you want to understand how real production-grade data pipelines are built from scratch, this video will give you a clear and practical understanding.
Subscribe to the channel for more Data Engineering, Big Data, and Cloud content.