Rack Awareness in Hadoop
Hello Achievers!!!
Welcome back to Softtechforum. In this video we have discussed the concept of Rack awareness in Hadoop.
HDFS stores files across multiple Data nodes in a cluster. To get maximum performance from Hadoop and to improve the network traffic during read/write, Name node chooses the Datanode on the same Rack or nearby Racks. Rack awareness is the concept of choosing the the closer Datanode based on the rack information.
We will discuss on below topics in this video :
What is Rack.
What is Rack Awareness in Hadoop.
Why Rack Awareness.
What is Rack ?
The Rack is the collection of 40-50 Data Nodes connected using the same network switch. If the network goes down, then the whole Rack will be unavailable. Large Hadoop cluster is deployed into multiple Racks.
What is Rack Awareness in Hadoop ?
In large Hadoop cluster, there are multiple racks. Each rack consists of DataNodes. Communication between the Data Nodes on the same rack is more efficient as compared to Data Nodes residing into different Racks.
To Reduce network traffic during the file Read/Write, Name Node chooses the nearest Data Nodes. Name Node maintains rack ids of each DataNode to achieve this rack information. The concept of choosing the nearest Datanode based on the rack information is known as Rack awareness.
Why Rack Awareness
To Reduce the network traffic during Read/Write, which improves the cluster performance.
To achieve fault tolerance even when the rack goes down.
To reduce the latency.
Name Node uses a rack awareness algorithm while placing the replicas in HDFS
Rack Awareness Policies
Not more than 1 replica be placed in one DataNode
Not more than 2 replica be placed in same rack
Number of rack used for the block replication should be always smaller than the Number of replicas
Hope you got the idea of Rack Awareness in Hadoop. Stay tuned for various other concepts.
Happy Learning!!!
Content Credit: https://data-flair.training/