25 AQE aka Adaptive Query Execution in Spark | Coalesce Shuffle Partitions | Skew Partitions Fix
Video explains - What is Adaptive Query Execution in Spark ? What is AQE? What Optimizations does AQE provides with Spark ?
Chapters
00:00 - Introduction
00:51 - What is AQE and What it Offers?
02:13 - Join without AQE
05:16 - Join with AQE
05:23 - AQE Coalesce Post Shuffle Partitions
05:48 - AQE Skew Partitions Optimization
09:14 - AQE SortMerge to BroadCast Join
Local PySpark Jupyter Lab setup - • 03 Setup Docker PySpark Jupyter Lab Enviro...
Python Basics - https://www.learnpython.org/
GitHub URL for code - https://github.com/subhamkharwal/pysp...
The series provides a step-by-step guide to learning PySpark, a popular open-source distributed computing framework that is used for big data processing.
New video in every 3 days ❤️
#spark #pyspark #python #dataengineering