Big Data, HDFS, Yarn And FileFormats
This video talks about HDFS concepts, brief overview of Yarn and various file formats (Parquet, Orc, Avro etc). available and where to use when.
Slides used in all the videos are - https://drive.google.com/file/d/1jNA3...
Subsequent videos cover below topics
1 - BigData HDFS Yarn And FileFormats : HDFS concepts, brief overview of Yarn and various file formats (Parquet, Orc, Avro etc)
2 - Spark basics, rdd, dataframe, dataset. Read from rdbms using Spark: Spark basics. What are rdd, dataframe and dataset. Shows an example to read from rdbms (MySQL) using Spark
3 - Spark - Dataframe / DataSet concepts, how to create these. Spark program, Spark job : This video discusses Dataframe / DataSet concepts, how to create these. How to write Spark program in IDE and what is Spark job.
4 - Spark Shuffle, RDD Functions, Partitions and related optimizations : This video discusses concepts and optimizations on Spark Shuffle and Partitions, RDD functions.
5 - Partitions, PartitionTypes, PartitionFunctions, RDD Types, Accumulator and BroadcastVariables: It describes partitions, partitions types and functions which can help you partition. Then it covers various RDD types with examples. Towards the end it discusses accumulators and broadcast variables.
6 - RDD Advanced Functions eg map, mapPartitions, fold, reduce etc : This video discusses various advance functions of RDD like map, mapPartitions, fold, reduce, foldByKey, reduceByKey etc. Some examples are taken from sparkbyexamples.com .
7 - Spark - Various Dataset Functions And Detailed Explanation Of Spark Plan : It discusses various dataset/dataframe functions eg min, max, describe, dropDuplicates, withColumn, when-otherwise condition etc.
Later part of the video discusses Spark plan in details.
8 - More Dataset functions like explode etc, tempViews, schema.Spark memory areas, configs etc : Spark More Dataset functions like explode, sort, unions, count, where, collect, sort, join, Later temp views are discussed. How Spark memory works, various memory configs, memory areas etc
9 - Apache Spark - Window Functions, SparkUI- what, how to look all info , RangePartitioner:
a) It starts with explaining Windows functions in Spark with examples.
b) Then it talks about Spark UI. What all info it has and how to look for each type of info.
c) After that it cover Range Partitioner. Explains what Range Partitioner is etc.
In the beginning, how to use off heap memory to persist Spark dataset is discussed. The article referred is - / spark-save-dataset-memory-outside-heap-san...
10 - Apache Spark Optimization - Optimizer Rules And How To Add Custom Rules : This video explains about Apache Spark optimizations. How Spark built in optimizations like Rule based, cost based etc work in details. And how can you add your own custom optimization rules.