Broadcast Join vs Shuffle Hash Join Explained | PySpark Join Strategies in Databricks
In this video, we will understand how joins happen internally in PySpark and Spark. You’ll learn about the two main join strategies — Broadcast Join and Shuffle Hash Join, how Spark decides which one to use, and how you can control this behavior in Databricks using configuration settings and hints.
All code and data files are available on the below path:
https://github.com/databeli/pyspark_c...
PowerPoint Presentaion useed in the complete playlist(27 slides)
https://topmate.io/narender_kumar_91/...
What you’ll learn:
Internal working of Spark joins
What is Broadcast Join and how it avoids data shuffling
How Shuffle Hash Join works when both datasets are large
Performance comparison: Broadcast Join vs Shuffle Join
Understanding data shuffling in Spark
Configure autoBroadcastJoinThreshold for custom tuning
Use spark.conf.set to adjust join size limits
Identify join type using explain() in PySpark
Use broadcast hints in PySpark and SQL
Learn how Databricks Photon Engine optimizes joins
#pyspark #pysparktutorial #databricks #databrickstutorial