Finance Domain End-to-End Data Engineering Project in Databricks (Free Edition) with Source Code
Welcome to this comprehensive, step-by-step tutorial on building an end-to-end Data Engineering project entirely within the Databricks Free Edition!
In this video, we will design and build a robust, metadata-driven framework from scratch. Whether you are looking to understand the core architecture of a data pipeline or want hands-on experience with the latest Databricks features, this tutorial covers it all—from data ingestion and transformation to orchestration, governance, and BI consumption.
Resources:
Notebooks used are available at below path:
https://github.com/databeli/databrick...
Plan 1 on 1 with me at below URL:
https://topmate.io/narender_kumar_91/
What You Will Learn:
00:00 Intro
02:09 Business Use Case
06:13 Architecture Deep Dive
11:42 Set up SQL Server as Source
15:20 Metadata Setup
20:24 Setup Secret Scope
29:11 Source to Bronze Ingestion
56:03 Bronze to Silver Ingestion
01:03:26 Creating Databricks Job
01:21:56 Databricks Volume Setup
01:31:39 Gold Layer Transformations
01:50:18 Send Email using Gmail SMTP Server
01:56:30 Create Databricks End to End Master Job
02:07:56 Creating Databricks Dashboard from Scratch
02:25:57 Set up Databricks Genie
02:30:19 Databricks One Demo
02:32:40 Conclusion and Next Steps
Tech Stack & Tools Covered:
Data Ingestion: Autoloader, PySpark, JDBC Connections
Transformation: Apache Spark, Spark SQL, Delta Lake, Databricks Notebooks
Orchestration: Databricks Workflows / Jobs
Governance & Security: Unity Catalog, Secret Scopes
Consumption: Databricks Dashboards, Databricks Genie Workspace, Databricks One
About the Instructor
I'm Narender Kumar! I have 12 years of experience in the IT industry, with the last 8 years heavily focused on Data Engineering. I hold multiple Architect-level certifications and am proud to be a recognized Databricks Solutions Architect Champion. My goal is to help you master data engineering concepts through real-world, practical projects.
Other related videos:
PySpark and Databricks Complete Course
• PySpark and Databricks Complete Course | B...
Databricks All in One Complete Course
• Databricks All in One Complete Course
Most Detailed Roadmap to Become Data Engineer
• Most Detailed Roadmap to Become Data Engin...
Data Engineer Resume | Step-by-Step Guide in Details
• Data Engineer Resume | Step-by-Step Guide ...
Don't forget to Like, Subscribe, and turn on notifications if you find this helpful! Let's get started.
#databricks #dataengineering #pyspark