How to Screw up your Repartitioning! - Spark Partitioning (Part 11)
In today's episode we will screw up repartitioning. On purpose of course 😉 By that we discover how the following Spark Partitioning and shuffling concepts work:
Round Robin Partitioning
Hash Partitioning
Range Partitioning
You want to master Data Engineering with PySpark? Subscribe here: https://www.youtube.com/@DataNikktheG...
Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.
Link to the code can be found here: https://github.com/datanikkthegreek/S...
00:00 - Intro
01:50 - Recap Repartition
03:30 - Round Robin Partitioning
11:15 - Hash Partitioning
18:55 - Range Partitioning
23:00 - Summary
#spark #pyspark #dataengineering #dataengineeringessentials