Optimize your Spark Executions with better Spark Partitions - Load Big Data Efficiently (Part 8)
In memory Partitions is one of the core concept in Spark. A bad distribution can lead to inefficient data execution within Spark. When loading data or the following transformations.
In this video we will have a look what influences partitions and how we can optimize partitions for more efficient data loading. Check it out :)
~~~~~~~~~~~~~ Subscribe - Like - Comment - Challenge ~~~~~~~~~~~~~
You want to master Data Engineering with PySpark? Subscribe here: https://www.youtube.com/@DataNikktheG...
Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.
~~~~~~~~~~~~~~~~~~~~~~~ Resources ~~~~~~~~~~~~~~~~~~~~~~~
Link to Slides: https://github.com/datanikkthegreek/S...
Link to code: https://github.com/datanikkthegreek/S...
All details about partitions: • Spark Partitioning
~~~~~~~~~~~~~~~~~~~~~~~ Chapters ~~~~~~~~~~~~~~~~~~~~~~~
00:00 - Introduction
00:45 - Partitions Theory
09:45 - Partitions in Action
29:50 - Summary
#spark #pyspark #dataengineering #dataengineeringessentials