Future of Data Engineering: 2025 & Beyond with Michael Armbrust
Join Michael Armbrust (Databricks) as he explores the evolution of data engineering from Hadoop to declarative Spark SQL and the future of coding in English with LLMs.
He provides a deep dive into Delta Live Tables (DLT), explaining its declarative approach, streaming tables, materialized views, and seamless integration with Serverless for optimized pipelines.
Discover DLT's role as the core engine for Databricks products, its improved developer experience, and the vision for Lake Flow and AI-driven data engineering
Chapters
• 0:00 - Introduction: The Evolution of Data Engineering at Databricks
• 1:30 - Deep Dive into Delta Live Tables (DLT): Declarative Data Engineering
• 3:30 - DLT Abstractions: Streaming Tables and Materialized Views
• 5:00 - Leveraging Serverless with DLT for Cost & Performance
• 6:30 - DLT as the Core Engine for Databricks Products
• 8:00 - Enhancing the DLT Developer Experience
• 9:30 - The Future Landscape: Open Source DLT, DBT, and Lake Flow
• 12:00 - The Role of AI in Data Engineering with Lake Flow Connect
• 14:30 - Conclusion: Embracing Declarative Data Engineering
Connect with the community:
🚀 LinkedIn Youssef Mrini: / youssefmrini
🚀 Subscribe to the NextGenLakehouse Newsletter: https://www.nextgenlakehouse.com/
👥 Join Databricks subreddit Community: / databricks
🐦 Follow us on X: https://x.com/YoussefMrini
📸 Follow us on Instagram: / nextgenlakehouse
🎥 Follow us on TikTok: / nextgenlakehouse
💻 Explore the Databricks Developer Hub: https://developers.databricks.com/