Day 9: Spark Lazy Evaluation Explained | Actions & Transformations in Apache Spark
🔥 Apache Spark Lazy Evaluation Explained! Learn how Spark optimizes transformations and executes only when an action is triggered. We’ll break down DAG (Directed Acyclic Graph), explain lazy evaluation, and cover all Spark transformations & actions with examples!
📌 Topics Covered in This Video:
✅ What is Lazy Evaluation?
✅ How Spark Executes Transformations?
✅ DAG (Directed Acyclic Graph) Explained
✅ Wide vs Narrow Transformations
✅ List of All Spark Transformations & Actions
🔹 Spark Transformations (Lazy - Do Not Execute Immediately)
Transformations return a new RDD/DataFrame but do not trigger execution.
✅ Narrow Transformations (No Shuffling Required)
1️⃣ map(func) – Applies a function to each element.
2️⃣ flatMap(func) – Similar to map but flattens the result.
3️⃣ filter(func) – Filters elements based on a condition.
4️⃣ distinct() – Removes duplicate elements.
5️⃣ union(rdd) – Combines two RDDs.
6️⃣ sample(withReplacement, fraction) – Extracts a fraction of data randomly.
✅ Wide Transformations (Shuffling Required)
7️⃣ groupByKey() – Groups elements by key (Key-Value RDD).
8️⃣ reduceByKey(func) – Merges values for each key using a function.
9️⃣ sortBy(func) – Sorts RDD based on the given function.
🔟 intersection(rdd) – Returns common elements between two RDDs.
1️⃣1️⃣ subtract(rdd) – Removes elements of another RDD.
1️⃣2️⃣ join(rdd) – Joins two RDDs based on keys.
1️⃣3️⃣ coalesce(numPartitions) – Reduces the number of partitions.
1️⃣4️⃣ repartition(numPartitions) – Shuffles data to increase partitions.
🔹 Spark Actions (Trigger Execution & Return Result)
Actions trigger execution of all previous transformations.
1️⃣ collect() – Returns all elements to the driver.
2️⃣ count() – Returns the total number of elements.
3️⃣ first() – Returns the first element.
4️⃣ take(n) – Returns the first n elements.
5️⃣ reduce(func) – Aggregates elements using a function.
6️⃣ foreach(func) – Applies a function to each element.
7️⃣ show(n) – Displays the first n rows (DataFrame).
8️⃣ write.format().save(path) – Saves data to storage.