Spark Skewed Data Problem: How to Fix it Like a Pro
Skewed data causes a lot of issues in Spark jobs, Learn how to deal with skewed data in Spark.
We will focus on one of the most important techniques for dealing with skewed data: Salting.
Content for the video:
00:00 Intro
00:36 Agenda
01:07 What is skewness
02:39 Why to worry about skewed data?
04:55 Different ways to handle skewed data
06:42 Salting technique
07:34 Salting coding demonstration
16:08 Benefits and Drawbacks of Salting Technique
Databricks notebook: https://databricks-prod-cloudfront.cl...
Apache Spark for everyone playlist: • StructType and StructField in PySpark | Sp...