Part 4: PySpark Transformations - Repartition and Coalesce
Connect with me here:
LinkedIn: www.linkedin.com/in/meenakshi-lakshmanan
Subscribe to my channel:
/ @conquerdatascience281
Welcome again to the Pyspark Transformations and Actions.
In this video let us continue to understand about other two important transformations namely repartition and Coalesce,
Repartition:
PySpark Repartition is a concept in PySpark that is used to increase or decrease the partitions used for processing the RDD/Data Frame in PySpark model.
Coalesce:
The Coalesce function reduces the number of partitions in the PySpark Data Frame. By reducing it avoids the full shuffle of data and shuffles the data using the hash partitioner; this is the default shuffling mechanism used for shuffling the data.