Working in Big Data? Understand Spark's main concepts now! - Spark Basics Simply Explained (Part 2)
You are working in Big Data and AI? You want to understand how Spark's main concepts work... and also become an expert? Then this video is for you!
After this you will know how Spark processes Big Data lightning fast and efficient for your analytics use-cases!
We will talk about:
Deep dive into architecture by observing a filter and count
Lazy Evaluation and Catalyst Optimizer which optimizes your code in the background
Actions and Transformations
Jobs, Stages, Tasks
Did you miss the previous video? Check it out here: • Working in Big Data? You should know about...
You want to master Data Engineering with PySpark? Subscribe here: https://www.youtube.com/@DataNikktheG...
Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.
Link to slides: https://github.com/datanikkthegreek/S...
00:00 - Intro
02:00 - Spark Architecture when using filter and write
05:20 - Spark Architecture when using count
08:11 - Lazy Evaluation and Catalyst Optimizer
15:54 - Actions and Transformations
18:28 - Jobs, Stages, Tasks
21:39 - Summary
#spark #pyspark #dataengineering #dataengineeringessentials