36 - Spark RDD Transformations - map(), flatMap(), filter()
@backstreetbrogrammer
--------------------------------------------------------------------------------
Chapter 07 - Spark RDD Transformations - map(), flatMap(), filter()
--------------------------------------------------------------------------------
RDD Transformations are Spark operations when executed on RDD, it results in a single or multiple new RDD’s. Since RDD are immutable in nature, transformations always create new RDD without updating an existing one.
RDD Transformations are lazy operations meaning none of the transformations get executed until we call an action on Spark RDD. Since RDD’s are immutable, any transformations on it result in a new RDD leaving the current one unchanged.
There are two types of RDD transformations:
1. Narrow Transformation: Transformations which compute data living on a single partition =: there will not be any data movement between partitions to execute narrow transformations. Functions such as map(), mapPartition(), flatMap(), filter(), union() are some examples of narrow transformation.
2. Wider Transformation: Transformations which compute data living on many partitions =: there will be data movements between partitions to execute wider transformations. Since these shuffles the data, they also called shuffle transformations. Functions such as groupByKey(), aggregateByKey(), aggregate(), join(), repartition() are some examples of a wider transformations.
When compared to Narrow transformations, wider transformations are expensive operations due to shuffling.
Github: https://github.com/backstreetbrogramm...
Apache Spark for Java Developers Playlist: • Apache Spark for Java Developers
Top Java Coding Interview Problems Playlist: • Top Java Coding Interview Problems
Java Serialization Playlist: • Java Serialization
Dynamic Programming Playlist: • Dynamic Programming
#java #javadevelopers #javaprogramming #apachespark #spark