Faster Data Loads with Spark - Load Big Data Efficiently (Part 9)
Already during the load of data a lot of inefficiencies can occur in Spark. A fast and efficient load of your data can improve your overall ETL pipeline. Let's have a look in a nutshell how we can make your data loading efficient. Check it out :)
In this video we will cover the following:
The choice of the file format matters. Parquet and Avro are more beneficial for big data processing
How the small file problem behaves. What are reasons and we can deal with it.
How defining the schema improves performance
How adjusting config openCostInBytes can improve the small file Problem
Leveraging predicate and aggregate pushdown can improve your performance
How to generate well distributed partitions using the config maxPartitionBytes
~~~~~~~~~~~~~ Subscribe - Like - Comment - Challenge ~~~~~~~~~~~~~
You want to master Data Engineering with PySpark? Subscribe here: / @datanikkthegreek
Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.
~~~~~~~~~~~~~~~~~~~~~~~ Resources ~~~~~~~~~~~~~~~~~~~~~~~
Link to Slides: https://github.com/datanikkthegreek/S...
Link all code samples: https://github.com/datanikkthegreek/S...
All videos regarding efficient data loading: • Load Big Data Efficiently
~~~~~~~~~~~~~~~~~~~~~~~ Chapters ~~~~~~~~~~~~~~~~~~~~~~~
00:00 - Introduction
00:46 - The File Format Matters
06:16 - The Small File Problem
10:17 - The Force of the Schema
13:41 - Open Cost Per Bytes for Small File Problem
17:19 - Predicate and Aggregate Pushdown
24:17 - Better Partitions with Max Partition Bytes
31:23 - Summary
#spark #pyspark #dataengineering #dataengineeringessentials