05 - Apache Spark for Java Developers - Hadoop and MapReduce
@backstreetbrogrammer
--------------------------------------------------------------------------------
Chapter 01 - Apache Spark for Java Developers - Hadoop and MapReduce
--------------------------------------------------------------------------------
Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.
Hadoop uses Hadoop Distributed File System (HDFS) which is a distributed, scalable, and portable file system written in Java for the Hadoop framework and allows user to work with large data sets. It also duplicates blocks of data for fault tolerance.
HDFS uses MapReduce which allows computations on that data.
HDFS uses blocks of data of default size 128 MB and replicates it multiple times to the slave nodes for fault tolerance.
MapReduce is a way of splitting a computational task to a distributed set of files such as HDFS. It consists of a Job Tracker at Master Node and multiple Task Trackers in the slave nodes. Job Tracker sends code to run on the Task Trackers. The Task Trackers allocate CPU and memory for the tasks and monitor the tasks on the worker nodes.
To summarize,
HDFS is used to distribute large data sets
MapReduce is used to distribute a computational task to a distributed data set
Github: https://github.com/backstreetbrogramm...
Apache Spark for Java Developers Playlist: • Apache Spark for Java Developers
Java Serialization Playlist: • Java Serialization
Dynamic Programming Playlist: • Dynamic Programming
#java #javadevelopers #javaprogramming #apachespark #spark