06 - Apache Spark for Java Developers - Spark Ecosystem
@backstreetbrogrammer
--------------------------------------------------------------------------------
Chapter 01 - Apache Spark for Java Developers - Spark Ecosystem
--------------------------------------------------------------------------------
Apache Spark is a multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.
Key Features:
1) SQL analytics using RDDs and SparkSQL
Execute fast, distributed ANSI SQL queries for dashboards and ad-hoc reporting. Runs faster than most data warehouses.
2) Machine learning using SparkML
Train machine learning algorithms on a laptop and use the same code to scale to fault-tolerant clusters of thousands of machines.
3) Batch/streaming data using Spark Streaming
Unify the processing of data in batches and real-time streaming.
Spark is a flexible alternative to MapReduce.
MapReduce requires files to be stored only in HDFS, while Spark can work on data stored in a variety of formats like HDFS, AWS S3, Cassandra, HBase etc.
Spark can perform operations up to 100X faster than MapReduce because MapReduce writes most of the data to disk after each map and reduce operation; however Spark keeps most of the data in memory after each transformation. Spark will write to disk only when the memory is full.
Github: https://github.com/backstreetbrogramm...
Apache Spark for Java Developers Playlist: • Apache Spark for Java Developers
Java Serialization Playlist: • Java Serialization
Dynamic Programming Playlist: • Dynamic Programming
#java #javadevelopers #javaprogramming #apachespark #spark