Google+ Badge

Saturday, 4 June 2016

DEVELOPMENTS AND PROGRESS IN APACHE HADOOP HDFS

About HDFS:
HDFS is a Java-based file system that provides scalable and reliable data storage, and it was designed to span large clusters of commodity servers , in other words it is a distributed Java-based  file system for storing large volumes of data.HDFS forms the management layer along with the YARN.HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks.By distributing storage and computation across many servers, the combined storage resource can grow linearly with demand while remaining economical at every amount of storage.HDFS entails following features which ensures high availability , fault tolerance , scalability and efficient storage of data.

Rack awareness : Takes node's physical allocation                                          for scheduling tasks.

Standby NameNode: Main component for providing                                            redundancy and high availability

Less Data Movement: Processing of tasks takes place in the physical node where the data resides , hence data movement is reduced ,increasing high aggregate bandwidth


Features of HDFS


PROGRESS IN HDFS:

  • HDFS version 2.3.0 provides centralized cache management heterogeneous storage, implementation of openstack swift and HTTP supports
  • Version 2.4.0 provides metadata compatibility, rolling upgrades (allows upgrading individual HDFS daemons).
  • Version 2.5.0 provides incremental data copy , extended attributes for accessing metadata.