Day 8: Simplifying Apache Spark Architecture
In this video, we delve into the intricacies of Apache Spark's architecture, breaking down its core components and their interactions.
Below are the terms you can remember, however video covered only overview, good to have the terms familiarized.
Driver Program: Manages the Spark application lifecycle and orchestrates distributed operations.
Cluster Manager: Oversees resource allocation across the cluster, ensuring efficient task distribution.
Worker Nodes: Machines in the cluster that execute tasks assigned by the driver.
Executors: Processes running on worker nodes that execute tasks and manage data storage during processing.
Resilient Distributed Datasets (RDDs): Immutable data structures enabling fault-tolerant, parallel computations.
Directed Acyclic Graph (DAG): Represents the sequence of computations, optimizing task execution.
Partitions: Logical divisions of data within RDDs, enabling parallel processing.
Jobs: Individual Spark actions triggered by user code.
Stages: Subdivisions of a job, determined by shuffle boundaries.
Tasks: Units of work sent to executors, corresponding to operations on partitions.
SparkSession: The unified entry point for Spark applications, introduced in Spark 2.0.
SparkContext: The entry point for Spark functionality, responsible for connecting to the cluster.
Transformations: Operations on RDDs that return a new RDD, such as map or filter.
Actions: Operations that trigger computation and return results, like collect or count.
Lineage: The history of transformations applied to an RDD, used for fault recovery.
Broadcast Variables: Read-only variables cached on each machine, enhancing efficiency.
Accumulators: Variables used for aggregating information across executors, typically for counters or sums.
Spark SQL: Module for structured data processing using DataFrames and SQL queries.
DataFrames: Distributed collections of data organized into named columns, similar to relational tables.
Datasets: Strongly-typed, distributed collections of data, providing type safety and object-oriented programming interface.
Spark Streaming: Enables processing of real-time data streams.
Structured Streaming: High-level API for stream processing, building on Spark SQL's DataFrame and Dataset abstractions.
MLlib: Spark's scalable machine learning library.
GraphX: Spark's API for graph and graph-parallel computation.