SlideShare una empresa de Scribd logo
1 de 64
Detailed Intro. to HDFS
           July 10, 2012
             Clay Jiang
    Big Data Engineering Team
           Hanborq Inc.
HDFS Intro.
• Overview

• HDFS Internal

• HDFS O&M, Tools

• HDFS Future




                           2
Overview


           3
What is HDFS?
• Hadoop Distributed FileSystem
• Good For:
   Large Files
   Streaming Data Access
• NOT For:
  x Lots of Small Files
  x Random Access
  x Low-latency Access


                                  4
Design of HDFS
• GFS-like
  – http://research.google.com/archive/gfs.html
• Master-slave design
  – Master
     • Single NameNode for managing FS meta
  – Slaves
     • Multiple DataNode s for storing data
  – One more:
     • SecondaryNameNode for checkpointing

                                                  5
HDFS Architecture
•




                        6
HDFS Storage
• HDFS Files are broken into Blocks
  – Basic unit of reading/writing like disk block
  – Default to 64MB, may be larger in product env.
  – Make HDFS good for large file & high throughput
• Block may have multiple Replicas
  – One block stored as multiple locations
  – Make HDFS storage fault tolerant



                                                      7
HDFS Storage




               8
HDFS Internal


                9
HDFS Internal

• NameNode

• SecondaryNameNode

• DataNode



                       10
NameNode
• Filesystem Meta
  – FSNames
  – FSName  Blocks
  – Block  Replicas
• Interact With
  – Client
  – DataNode
  – SecondaryNameNode


                         11
NameNode FS Meta
• FSImage
  – FSNames & FSName  Blocks
  – Saved replicas in multiple name directory
  – Recover on startup
• EditLog
  – Log every FS modification
• Block  Replicas (DataNodes)
  – Only in memory
  – rebuilt from block reports on startup


                                                12
NameNode Interface
• Through different protocol interface
  – ClientProtocol:
     • Create, addBlock, delete, rename, fsync …
  – NameNodeProtocol:
     • rollEditLog, rollFsImage, …
  – DataNodeProtocol:
     • sendHeartbeat, blockReceived, blockReport, …
  –…


                                                      13
NameNode Startup
• On Startup
  – Load fsimage
  – Check safe mode
  – Start Daemons
     •   HeartbeatMonitor
     •   LeaseManager
     •   ReplicationMonitor
     •   DecommissionManager
  – Start RPC services
  – Start HTTP info server
  – Start Trash Emptier

                               14
Load FSImage
• Name Directory
  – dfs.name.dir: can be multiple dirs



  –       Check consistence of all name dirs
  –       Load fsimage file
  –       Load edit logs
  –       Save namespace
      •     Mainly setup dirs & files properly

                                                 15
Check Safemode
•   Safemode
    – Fsimage loaded but locations of blocks not known
      yet!
    – Exit when minimal replication condition meet
      •   dfs.safemode.threshold.pct
      •   dfs.replication.min
      •   Default case: 99.9% of block have 1 replicas
    – Start SafeModeMonitor to periodically check to
      leave safe mode
    – Leave safe mode manually
      •   hadoop dfsadmin -safemode leave
      •   (or enter it /get status by: hadoop dfsadmin -safemode
          enter/get)

                                                                   16
Start Daemons
• HeartbeatMonitor
  – Check lost DN & schedule necessary replication
• LeaseManager
  – Check lost lease
• ReplicationMonitor
  – computeReplicationWork
  – computeInvalidateWork
  – dfs.replication.interval, defautl to 3 secs
• DecommissionManager
  – Check and set node decommissioned

                                                     17
Trash Emptier
• /user/{user.name}/.Trash
  – fs.trash.interval > 0 to enable
  – When delete, file moved to .Trash
• Trash.Empiter
  – Run every fs.trash.interval mins
  – Delete old checkpoint (fs.trash interval mins ago)




                                                         18
HDFS Internal

• NameNode

• SecondaryNameNode

• DataNode



                       19
SecondaryNameNode
• Not Standby/Backup NameNode
  – Only for checkpointing
  – Though, has a NON-Realtime copy of FSImage
• Need as much memory as NN to do the
  checkpointing
  – Estimation: 1GB for every one million blocks




                                                   20
SecondaryNameNode
• Do the checkpointing
   – Copy NN’s fsimage &
     editlogs
   – Merge them to a new
     fsimage
   – Replace NN’s fsimage with
     new one & clean editlogs
• Timing
   – Size of editlog >
     fs.checkpoint.size (poll
     every 5 min)
   – Every fs.checkpoint.period
     secs

                                  21
HDFS Internal

• NameNode

• SecondaryNameNode

• DataNode



                       22
DataNode
• Store data blocks
  – Have no knowledge about FSName
• Receive blocks from Client
• Receive blocks from DataNode peer
  – Replication
  – Pipeline writing
• Receive delete command from NameNode


                                         23
Block Placement Policy
On Cluster Level
• replication = 3
  – First replica local
    with Client
  – Second & Third
    on two nodes of
    same remote rack




                             24
Block Placement Policy
On one single node
• Write each disk in turn
  – No balancing is considered !
• Skip a disk when it’s almost full or failed
• DataNode may go offline when disks failed
  – dfs.datanode.failed.volumes.tolerated




                                                25
DataNode Startup
• On DN Startup:
  –   Load data dirs
  –   Register itself to NameNode
  –   Start IPC Server
  –   Start DataXceiverServer
       • Transfer blocks
  – Run the main loop …
       •   Start BlockScanner
       •   Send heartbeats
       •   Process command from NN
       •   Send block report

                                     26
DataXceiverServer
• Accept data connection & start DataXceiver
  – Max num: dfs.datanode.max.xcievers [256]
• DataXceiver
  – Handle blocks
     •   Read block
     •   Write block
     •   Replace block
     •   Copy block
     •   …


                                               27
HDFS Routines Analysis

• Write File

• Read File

• Decrease Replication Factor

• One DN down

                                28
Write File
• Sample Code:
  DFSClient dfsclient = …;
  outputStream = dfsclient.create(…);
  outputStream.write(someBytes);
  …
  outputStream.close();
  dfsclient.close();



                                        29
Write File




             30
Write File
• DFSClient.create
  – NameNode.create
     •   Check existence
     •   Check permission
     •   Check and get Lease
     •   Add new INode to rootDir




                                    31
Write File
• outputStream.write
  – Get DNs to write to From NN
  – Break bytes into packets
  – Write packets to First DataNode’s DataXceiver
  – DN mirror packet to downstream DNs (Pipeline)
  – When complete, confirm NN blockReceived




                                                    32
Write File
• outputStream.close
  – NameNode.complete
     • Remove lease

     • Change file from “under construction” to “complete”




                                                             33
Lease




        34
Lease
• What is lease ?
  – Write lock for file modification

  – No lease for reading files

• Avoid concurrent write on the same file
  – Cause inconsistent & undefined behavior



                                              35
Lease
• LeaseManager
  – Lease is managed in NN
  – When file create (or append), lease added
• DFSClient.LeaseChecker
  – Client start thread to renew lease periodically




                                                      36
Lease Expiration
• Soft Limit
  – No renewing for 1 min
  – Other client compete for the lease
• Hard Limit
  – No renewing for 60 min (60 * softLimit)
  – No competition for the lease




                                              37
Read File
• Sample Code:
  DFSClient dfsclient = …
  FSDataInputStream is = dfsclient.open(…)
  is.read(…)
  is.close()




                                             38
Read File




            39
Read File
• DFClient.open
  – Create FSDataInputStream
     • Get block locations of file from NN
• FSDataInputStream.read
  – Read data from DNs block by block
     • Read the data
     • Do the checksum




                                             40
Desc Repl
• Code Sample
  DFSClient dfsclient = …;
  dfsclient.setReplication(…, 2) ;
• Or use the CLI
  hadoop fs -setrep -w 2 /path/to/file




                                         41
Desc Repl
•




                42
Desc Repl
• Change FSName replication factor
• Choose excess replicas
  – Rack number do not decrease
  – Get block from least available disk space node
• Add to invalidateSets(to-be-deleted block set)
• ReplicationMonitor compute blocks to be deleted
  for each DN
• On next DN’s heartbeat, give delete block
  command to DN
• DN delete specified blocks
• Update blocksMap when DN send blockReport
                                                     43
One DN down
• DataNode stop sending heartbeat
• NameNode
  – HeartbeatMonitor find DN dead when doing heartbeat
    check
  – Remove all blocks belong to DN
  – Update neededReplications (block set need one or more
    replication)
  – ReplicationMonitor compute block to be replicated for
    each DN
  – On next DN’s heartbeat, NameNode send replication block
    command
• DataNode
  – Replicate block

                                                          44
O&M, Tools


             45
High Availability
• NameNode SPOF
  – NameNode hold all the meta
  – If NN crash, the whole cluster unavailable
• Though fsimage can recover from SNN
  – It’s not a up-to-date fsimage
• Need HA solutions




                                                 46
HA Solutions

• DRBD

• Avatar Node

• Backup Node



                        47
HA - DRBD
• DRBD (http://www.drbd.org)
   – block devices designed as a building block to form
     high availability (HA) clusters.
   – Like network based raid-1
• Use DRBD to backup NN’s fsimage & editlogs
   – A cold backup for NN
   – Restart NN cost no more than 10 minutes



                                                      48
HA - DRBD
• Mirror one of NN’s name dir to a remote node
  – All name dir is the same
• When NN fail
  – Copy mirrored name dir to all name dir
  – Restart NN
  – All will be done in no more than 20 mins




                                               49
HA Solutions

• DRBD

• Avatar Node

• Backup Node



                        50
HA - AvatarNode
• Complete Hot Standby
  – NFS for storage of fsimage and editlogs
  – Standby node Consumes transactions from
    editlogs on NFS continuously
  – DataNodes send message to both primary and
    standby node
• Fast Switchover
  – Less than a minute


                                                 51
HA - AvatarNode
• Active-Standby Pair                                       Client
   – Coordinated via ZooKeeper
   – Failover in few seconds                           Client retrieves block
                                                       location from Primary
   – Wrapper over NameNode                             or Standby


• Active AvatarNode                     Active
                                                    Write
                                                    transaction
                                                                   Read
                                                                   transaction     Standby
                                      AvatarNode
   – Writes transaction log to NFS                                                AvatarNode
                                     (NameNode)            NFS                   (NameNode)
     filer
                                                           Filer
• Standby AvatarNode
                                         Block                                   Block
   – Reads/Consumes                      Location                                Location
     transactions from NFS filer         messages                                messages
   – Processes all messages from
     DataNodes                                         DataNodes

   – Latest metadata in memory
                                                                                         52
HA - AvatarNode
• Four steps to failover
   – Wipe ZooKeeper entry. Clients will know the failover
     is in progress. (0 seconds)
   – Stop the primary NameNode. Last bits of data will be
     flushed to Transaction Log and it will die. (Seconds)
   – Switch Standby to Primary. It will consume the rest of
     the Transaction log and get out of SafeMode ready to
     serve traffic. (Seconds)
   – Update the entry in ZooKeeper. All the clients waiting
     for failover will pick up the new connection (0 seconds)
• After: Start the first node in the Standby Mode
   – Takes a while, but the cluster is up and running

                                                           53
HA - AvatarNode




                  54
HA Solutions

• DRBD

• Avatar Node

• Backup Node



                        55
HA - BackupNode
• NN synchronously
  streams                                    Client

  transaction log to                    Client retrieves block location from
  BackupNode                            NN

• BackupNode applies log          NN
                                                  Synchronous stream
                                                  transacton logs to
  to in-memory and disk       (NameNode)          BN

  image
                                                               BN
• BN always commit to disk               Block
                                         Location         (BackupNode)
                                         messages
  before success to NN
• If BN restarts, it has to
   catch up with NN
                                 DataNodes

                                                                           56
Tools
• More Tools …
  – Balancer

  – Fsck

  – Distcp




                         57
Tools - Balancer
• Need Re-Balancing
   – When new node is add to cluster
• bin/start-balancer.sh
   – Move block from over-utilized node to under-utilized node
• dfs.balance.bandwidthPerSec
   – Control the impact on business
• -t <threshold>
   – Default 10%
   – stop when difference from average utilization is less than
     threshold

                                                                  58
Tools - Fsck
• hadoop fsck /path/to/file
• Check HDFS’s healthy
  – Missing blocks, corrupt blocks, mis-replicated
    blocks …
• Get blocks & locations of files
  – hadoop fsck /path/to/file -files -blocks -locations



                                                          59
Tools - Distcp
• Inter-cluster copy
  – hadoop distcp -i –pp -log /logdir
    hdfs://srcip/srcpath/ /destpath
  – Use map-reduce(actually maps) to start a
    distributed-fashion copy
• Also fast copy in the same cluster



                                               60
HDFS Future


              61
Hadoop Future
• Short-circuit local reads
    – dfs.client.read.shortcircuit = true
    – Available in hadoop-1.x or cdh3u4
•   Native checksums (HDFS-2080)
•   BlockReader keepalive to DN (HDFS-941)
•   “Zero-copy read” support (HDFS-3051)
•   NN HA (HDFS-3042)
•   HDFS Federation
•   HDFS RAID
                                             62
References
• Tom White, Hadoop The definitive guide
• http://hadoop.apache.org/hdfs/
• Hadoop WiKi – HDFS
   – http://wiki.apache.org/hadoop/HDFS
• Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung, The
  Google File System
   – http://research.google.com/archive/gfs.html
• Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert
  Chansler , The Hadoop Distributed File System
   – http://storageconference.org/2010/Papers/MSST/Shvachko.pdf


                                                              63
The End
Thank You Very Much!
     chiangbing@gmail.com




                            64

Más contenido relacionado

La actualidad más candente

Introduction to hadoop and hdfs
Introduction to hadoop and hdfsIntroduction to hadoop and hdfs
Introduction to hadoop and hdfsshrey mehrotra
 
Hadoop HDFS NameNode HA
Hadoop HDFS NameNode HAHadoop HDFS NameNode HA
Hadoop HDFS NameNode HAHanborq Inc.
 
Hadoop HDFS Architeture and Design
Hadoop HDFS Architeture and DesignHadoop HDFS Architeture and Design
Hadoop HDFS Architeture and Designsudhakara st
 
Hadoop Distributed File System(HDFS) : Behind the scenes
Hadoop Distributed File System(HDFS) : Behind the scenesHadoop Distributed File System(HDFS) : Behind the scenes
Hadoop Distributed File System(HDFS) : Behind the scenesNitin Khattar
 
Hadoop HDFS by rohitkapa
Hadoop HDFS by rohitkapaHadoop HDFS by rohitkapa
Hadoop HDFS by rohitkapakapa rohit
 
Storage Systems for big data - HDFS, HBase, and intro to KV Store - Redis
Storage Systems for big data - HDFS, HBase, and intro to KV Store - RedisStorage Systems for big data - HDFS, HBase, and intro to KV Store - Redis
Storage Systems for big data - HDFS, HBase, and intro to KV Store - RedisSameer Tiwari
 
Hadoop Interacting with HDFS
Hadoop Interacting with HDFSHadoop Interacting with HDFS
Hadoop Interacting with HDFSApache Apex
 
Hadoop World 2011: HDFS Federation - Suresh Srinivas, Hortonworks
Hadoop World 2011: HDFS Federation - Suresh Srinivas, HortonworksHadoop World 2011: HDFS Federation - Suresh Srinivas, Hortonworks
Hadoop World 2011: HDFS Federation - Suresh Srinivas, HortonworksCloudera, Inc.
 
HDFS Trunncate: Evolving Beyond Write-Once Semantics
HDFS Trunncate: Evolving Beyond Write-Once SemanticsHDFS Trunncate: Evolving Beyond Write-Once Semantics
HDFS Trunncate: Evolving Beyond Write-Once SemanticsDataWorks Summit
 
Coordinating Metadata Replication: Survival Strategy for Distributed Systems
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsCoordinating Metadata Replication: Survival Strategy for Distributed Systems
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsKonstantin V. Shvachko
 
Introduction to HDFS and MapReduce
Introduction to HDFS and MapReduceIntroduction to HDFS and MapReduce
Introduction to HDFS and MapReduceUday Vakalapudi
 
Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File SystemAnand Kulkarni
 
Snapshot in Hadoop Distributed File System
Snapshot in Hadoop Distributed File SystemSnapshot in Hadoop Distributed File System
Snapshot in Hadoop Distributed File SystemBhavesh Padharia
 
Hadoop distributed file system
Hadoop distributed file systemHadoop distributed file system
Hadoop distributed file systemAnshul Bhatnagar
 

La actualidad más candente (20)

Introduction to hadoop and hdfs
Introduction to hadoop and hdfsIntroduction to hadoop and hdfs
Introduction to hadoop and hdfs
 
Hadoop HDFS NameNode HA
Hadoop HDFS NameNode HAHadoop HDFS NameNode HA
Hadoop HDFS NameNode HA
 
Hadoop HDFS Architeture and Design
Hadoop HDFS Architeture and DesignHadoop HDFS Architeture and Design
Hadoop HDFS Architeture and Design
 
Hadoop Distributed File System(HDFS) : Behind the scenes
Hadoop Distributed File System(HDFS) : Behind the scenesHadoop Distributed File System(HDFS) : Behind the scenes
Hadoop Distributed File System(HDFS) : Behind the scenes
 
Hadoop introduction
Hadoop introductionHadoop introduction
Hadoop introduction
 
Anatomy of file write in hadoop
Anatomy of file write in hadoopAnatomy of file write in hadoop
Anatomy of file write in hadoop
 
Hadoop HDFS by rohitkapa
Hadoop HDFS by rohitkapaHadoop HDFS by rohitkapa
Hadoop HDFS by rohitkapa
 
Storage Systems for big data - HDFS, HBase, and intro to KV Store - Redis
Storage Systems for big data - HDFS, HBase, and intro to KV Store - RedisStorage Systems for big data - HDFS, HBase, and intro to KV Store - Redis
Storage Systems for big data - HDFS, HBase, and intro to KV Store - Redis
 
Hdfs architecture
Hdfs architectureHdfs architecture
Hdfs architecture
 
Hadoop Interacting with HDFS
Hadoop Interacting with HDFSHadoop Interacting with HDFS
Hadoop Interacting with HDFS
 
Hadoop World 2011: HDFS Federation - Suresh Srinivas, Hortonworks
Hadoop World 2011: HDFS Federation - Suresh Srinivas, HortonworksHadoop World 2011: HDFS Federation - Suresh Srinivas, Hortonworks
Hadoop World 2011: HDFS Federation - Suresh Srinivas, Hortonworks
 
HDFS Trunncate: Evolving Beyond Write-Once Semantics
HDFS Trunncate: Evolving Beyond Write-Once SemanticsHDFS Trunncate: Evolving Beyond Write-Once Semantics
HDFS Trunncate: Evolving Beyond Write-Once Semantics
 
Coordinating Metadata Replication: Survival Strategy for Distributed Systems
Coordinating Metadata Replication: Survival Strategy for Distributed SystemsCoordinating Metadata Replication: Survival Strategy for Distributed Systems
Coordinating Metadata Replication: Survival Strategy for Distributed Systems
 
Introduction to HDFS and MapReduce
Introduction to HDFS and MapReduceIntroduction to HDFS and MapReduce
Introduction to HDFS and MapReduce
 
Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File System
 
March 2011 HUG: HDFS Federation
March 2011 HUG: HDFS FederationMarch 2011 HUG: HDFS Federation
March 2011 HUG: HDFS Federation
 
Hadoop Architecture
Hadoop ArchitectureHadoop Architecture
Hadoop Architecture
 
Snapshot in Hadoop Distributed File System
Snapshot in Hadoop Distributed File SystemSnapshot in Hadoop Distributed File System
Snapshot in Hadoop Distributed File System
 
Hadoop distributed file system
Hadoop distributed file systemHadoop distributed file system
Hadoop distributed file system
 
HDFS Internals
HDFS InternalsHDFS Internals
HDFS Internals
 

Similar a Hadoop HDFS Detailed Introduction

Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File SystemRutvik Bapat
 
Snapshots In Red Hat Storage Server Overview & Quickstart
Snapshots In Red Hat Storage Server Overview & QuickstartSnapshots In Red Hat Storage Server Overview & Quickstart
Snapshots In Red Hat Storage Server Overview & QuickstartRed_Hat_Storage
 
File service architecture and network file system
File service architecture and network file systemFile service architecture and network file system
File service architecture and network file systemSukhman Kaur
 
Introduction to distributed file systems
Introduction to distributed file systemsIntroduction to distributed file systems
Introduction to distributed file systemsViet-Trung TRAN
 
Infrastructure Around Hadoop
Infrastructure Around HadoopInfrastructure Around Hadoop
Infrastructure Around HadoopDataWorks Summit
 
Ericas-Linux-Plus-Study-Guide
Ericas-Linux-Plus-Study-GuideEricas-Linux-Plus-Study-Guide
Ericas-Linux-Plus-Study-GuideErica StJohn
 
Backup, Restore, and Disaster Recovery
Backup, Restore, and Disaster RecoveryBackup, Restore, and Disaster Recovery
Backup, Restore, and Disaster RecoveryMongoDB
 
Deployment Strategy
Deployment StrategyDeployment Strategy
Deployment StrategyMongoDB
 
Big Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosBig Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosHeiko Loewe
 
Resource Management of Docker
Resource Management of DockerResource Management of Docker
Resource Management of DockerSpeedyCloud
 
Hadoop - Disk Fail In Place (DFIP)
Hadoop - Disk Fail In Place (DFIP)Hadoop - Disk Fail In Place (DFIP)
Hadoop - Disk Fail In Place (DFIP)mundlapudi
 
Gfs google-file-system-13331
Gfs google-file-system-13331Gfs google-file-system-13331
Gfs google-file-system-13331Fengchang Xie
 
Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)MongoDB
 
001 linux revision
001 linux revision001 linux revision
001 linux revisionSherif Mousa
 

Similar a Hadoop HDFS Detailed Introduction (20)

Hadoop Distributed File System
Hadoop Distributed File SystemHadoop Distributed File System
Hadoop Distributed File System
 
A32 Database Virtulization Technologies
A32 Database Virtulization TechnologiesA32 Database Virtulization Technologies
A32 Database Virtulization Technologies
 
Snapshots In Red Hat Storage Server Overview & Quickstart
Snapshots In Red Hat Storage Server Overview & QuickstartSnapshots In Red Hat Storage Server Overview & Quickstart
Snapshots In Red Hat Storage Server Overview & Quickstart
 
Big data- HDFS(2nd presentation)
Big data- HDFS(2nd presentation)Big data- HDFS(2nd presentation)
Big data- HDFS(2nd presentation)
 
File service architecture and network file system
File service architecture and network file systemFile service architecture and network file system
File service architecture and network file system
 
HDFS Basics
HDFS BasicsHDFS Basics
HDFS Basics
 
Introduction to distributed file systems
Introduction to distributed file systemsIntroduction to distributed file systems
Introduction to distributed file systems
 
Infrastructure Around Hadoop
Infrastructure Around HadoopInfrastructure Around Hadoop
Infrastructure Around Hadoop
 
Ericas-Linux-Plus-Study-Guide
Ericas-Linux-Plus-Study-GuideEricas-Linux-Plus-Study-Guide
Ericas-Linux-Plus-Study-Guide
 
2 technical-dns-workshop-day1
2 technical-dns-workshop-day12 technical-dns-workshop-day1
2 technical-dns-workshop-day1
 
Backup, Restore, and Disaster Recovery
Backup, Restore, and Disaster RecoveryBackup, Restore, and Disaster Recovery
Backup, Restore, and Disaster Recovery
 
Deployment Strategy
Deployment StrategyDeployment Strategy
Deployment Strategy
 
Big Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and MesosBig Data in Container; Hadoop Spark in Docker and Mesos
Big Data in Container; Hadoop Spark in Docker and Mesos
 
Resource Management of Docker
Resource Management of DockerResource Management of Docker
Resource Management of Docker
 
Hadoop - Disk Fail In Place (DFIP)
Hadoop - Disk Fail In Place (DFIP)Hadoop - Disk Fail In Place (DFIP)
Hadoop - Disk Fail In Place (DFIP)
 
Gfs google-file-system-13331
Gfs google-file-system-13331Gfs google-file-system-13331
Gfs google-file-system-13331
 
Hadoop admin
Hadoop adminHadoop admin
Hadoop admin
 
2_Chapter 2_DNS.pptx
2_Chapter 2_DNS.pptx2_Chapter 2_DNS.pptx
2_Chapter 2_DNS.pptx
 
Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)Deployment Strategies (Mongo Austin)
Deployment Strategies (Mongo Austin)
 
001 linux revision
001 linux revision001 linux revision
001 linux revision
 

Más de Hanborq Inc.

Introduction to Cassandra
Introduction to CassandraIntroduction to Cassandra
Introduction to CassandraHanborq Inc.
 
Hadoop大数据实践经验
Hadoop大数据实践经验Hadoop大数据实践经验
Hadoop大数据实践经验Hanborq Inc.
 
Flume and Flive Introduction
Flume and Flive IntroductionFlume and Flive Introduction
Flume and Flive IntroductionHanborq Inc.
 
Hadoop MapReduce Streaming and Pipes
Hadoop MapReduce  Streaming and PipesHadoop MapReduce  Streaming and Pipes
Hadoop MapReduce Streaming and PipesHanborq Inc.
 
HBase Introduction
HBase IntroductionHBase Introduction
HBase IntroductionHanborq Inc.
 
Hadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler IntroductionHadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler IntroductionHanborq Inc.
 
Hadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep InsightHadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep InsightHanborq Inc.
 
How to Build Cloud Storage Service Systems
How to Build Cloud Storage Service SystemsHow to Build Cloud Storage Service Systems
How to Build Cloud Storage Service SystemsHanborq Inc.
 
Hanborq Optimizations on Hadoop MapReduce
Hanborq Optimizations on Hadoop MapReduceHanborq Optimizations on Hadoop MapReduce
Hanborq Optimizations on Hadoop MapReduceHanborq Inc.
 

Más de Hanborq Inc. (11)

Introduction to Cassandra
Introduction to CassandraIntroduction to Cassandra
Introduction to Cassandra
 
Hadoop大数据实践经验
Hadoop大数据实践经验Hadoop大数据实践经验
Hadoop大数据实践经验
 
FlumeBase Study
FlumeBase StudyFlumeBase Study
FlumeBase Study
 
Flume and Flive Introduction
Flume and Flive IntroductionFlume and Flive Introduction
Flume and Flive Introduction
 
Hadoop MapReduce Streaming and Pipes
Hadoop MapReduce  Streaming and PipesHadoop MapReduce  Streaming and Pipes
Hadoop MapReduce Streaming and Pipes
 
HBase Introduction
HBase IntroductionHBase Introduction
HBase Introduction
 
Hadoop Versioning
Hadoop VersioningHadoop Versioning
Hadoop Versioning
 
Hadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler IntroductionHadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler Introduction
 
Hadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep InsightHadoop MapReduce Introduction and Deep Insight
Hadoop MapReduce Introduction and Deep Insight
 
How to Build Cloud Storage Service Systems
How to Build Cloud Storage Service SystemsHow to Build Cloud Storage Service Systems
How to Build Cloud Storage Service Systems
 
Hanborq Optimizations on Hadoop MapReduce
Hanborq Optimizations on Hadoop MapReduceHanborq Optimizations on Hadoop MapReduce
Hanborq Optimizations on Hadoop MapReduce
 

Último

IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 

Último (20)

IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 

Hadoop HDFS Detailed Introduction

  • 1. Detailed Intro. to HDFS July 10, 2012 Clay Jiang Big Data Engineering Team Hanborq Inc.
  • 2. HDFS Intro. • Overview • HDFS Internal • HDFS O&M, Tools • HDFS Future 2
  • 4. What is HDFS? • Hadoop Distributed FileSystem • Good For:  Large Files  Streaming Data Access • NOT For: x Lots of Small Files x Random Access x Low-latency Access 4
  • 5. Design of HDFS • GFS-like – http://research.google.com/archive/gfs.html • Master-slave design – Master • Single NameNode for managing FS meta – Slaves • Multiple DataNode s for storing data – One more: • SecondaryNameNode for checkpointing 5
  • 7. HDFS Storage • HDFS Files are broken into Blocks – Basic unit of reading/writing like disk block – Default to 64MB, may be larger in product env. – Make HDFS good for large file & high throughput • Block may have multiple Replicas – One block stored as multiple locations – Make HDFS storage fault tolerant 7
  • 10. HDFS Internal • NameNode • SecondaryNameNode • DataNode 10
  • 11. NameNode • Filesystem Meta – FSNames – FSName  Blocks – Block  Replicas • Interact With – Client – DataNode – SecondaryNameNode 11
  • 12. NameNode FS Meta • FSImage – FSNames & FSName  Blocks – Saved replicas in multiple name directory – Recover on startup • EditLog – Log every FS modification • Block  Replicas (DataNodes) – Only in memory – rebuilt from block reports on startup 12
  • 13. NameNode Interface • Through different protocol interface – ClientProtocol: • Create, addBlock, delete, rename, fsync … – NameNodeProtocol: • rollEditLog, rollFsImage, … – DataNodeProtocol: • sendHeartbeat, blockReceived, blockReport, … –… 13
  • 14. NameNode Startup • On Startup – Load fsimage – Check safe mode – Start Daemons • HeartbeatMonitor • LeaseManager • ReplicationMonitor • DecommissionManager – Start RPC services – Start HTTP info server – Start Trash Emptier 14
  • 15. Load FSImage • Name Directory – dfs.name.dir: can be multiple dirs – Check consistence of all name dirs – Load fsimage file – Load edit logs – Save namespace • Mainly setup dirs & files properly 15
  • 16. Check Safemode • Safemode – Fsimage loaded but locations of blocks not known yet! – Exit when minimal replication condition meet • dfs.safemode.threshold.pct • dfs.replication.min • Default case: 99.9% of block have 1 replicas – Start SafeModeMonitor to periodically check to leave safe mode – Leave safe mode manually • hadoop dfsadmin -safemode leave • (or enter it /get status by: hadoop dfsadmin -safemode enter/get) 16
  • 17. Start Daemons • HeartbeatMonitor – Check lost DN & schedule necessary replication • LeaseManager – Check lost lease • ReplicationMonitor – computeReplicationWork – computeInvalidateWork – dfs.replication.interval, defautl to 3 secs • DecommissionManager – Check and set node decommissioned 17
  • 18. Trash Emptier • /user/{user.name}/.Trash – fs.trash.interval > 0 to enable – When delete, file moved to .Trash • Trash.Empiter – Run every fs.trash.interval mins – Delete old checkpoint (fs.trash interval mins ago) 18
  • 19. HDFS Internal • NameNode • SecondaryNameNode • DataNode 19
  • 20. SecondaryNameNode • Not Standby/Backup NameNode – Only for checkpointing – Though, has a NON-Realtime copy of FSImage • Need as much memory as NN to do the checkpointing – Estimation: 1GB for every one million blocks 20
  • 21. SecondaryNameNode • Do the checkpointing – Copy NN’s fsimage & editlogs – Merge them to a new fsimage – Replace NN’s fsimage with new one & clean editlogs • Timing – Size of editlog > fs.checkpoint.size (poll every 5 min) – Every fs.checkpoint.period secs 21
  • 22. HDFS Internal • NameNode • SecondaryNameNode • DataNode 22
  • 23. DataNode • Store data blocks – Have no knowledge about FSName • Receive blocks from Client • Receive blocks from DataNode peer – Replication – Pipeline writing • Receive delete command from NameNode 23
  • 24. Block Placement Policy On Cluster Level • replication = 3 – First replica local with Client – Second & Third on two nodes of same remote rack 24
  • 25. Block Placement Policy On one single node • Write each disk in turn – No balancing is considered ! • Skip a disk when it’s almost full or failed • DataNode may go offline when disks failed – dfs.datanode.failed.volumes.tolerated 25
  • 26. DataNode Startup • On DN Startup: – Load data dirs – Register itself to NameNode – Start IPC Server – Start DataXceiverServer • Transfer blocks – Run the main loop … • Start BlockScanner • Send heartbeats • Process command from NN • Send block report 26
  • 27. DataXceiverServer • Accept data connection & start DataXceiver – Max num: dfs.datanode.max.xcievers [256] • DataXceiver – Handle blocks • Read block • Write block • Replace block • Copy block • … 27
  • 28. HDFS Routines Analysis • Write File • Read File • Decrease Replication Factor • One DN down 28
  • 29. Write File • Sample Code: DFSClient dfsclient = …; outputStream = dfsclient.create(…); outputStream.write(someBytes); … outputStream.close(); dfsclient.close(); 29
  • 31. Write File • DFSClient.create – NameNode.create • Check existence • Check permission • Check and get Lease • Add new INode to rootDir 31
  • 32. Write File • outputStream.write – Get DNs to write to From NN – Break bytes into packets – Write packets to First DataNode’s DataXceiver – DN mirror packet to downstream DNs (Pipeline) – When complete, confirm NN blockReceived 32
  • 33. Write File • outputStream.close – NameNode.complete • Remove lease • Change file from “under construction” to “complete” 33
  • 34. Lease 34
  • 35. Lease • What is lease ? – Write lock for file modification – No lease for reading files • Avoid concurrent write on the same file – Cause inconsistent & undefined behavior 35
  • 36. Lease • LeaseManager – Lease is managed in NN – When file create (or append), lease added • DFSClient.LeaseChecker – Client start thread to renew lease periodically 36
  • 37. Lease Expiration • Soft Limit – No renewing for 1 min – Other client compete for the lease • Hard Limit – No renewing for 60 min (60 * softLimit) – No competition for the lease 37
  • 38. Read File • Sample Code: DFSClient dfsclient = … FSDataInputStream is = dfsclient.open(…) is.read(…) is.close() 38
  • 39. Read File 39
  • 40. Read File • DFClient.open – Create FSDataInputStream • Get block locations of file from NN • FSDataInputStream.read – Read data from DNs block by block • Read the data • Do the checksum 40
  • 41. Desc Repl • Code Sample DFSClient dfsclient = …; dfsclient.setReplication(…, 2) ; • Or use the CLI hadoop fs -setrep -w 2 /path/to/file 41
  • 43. Desc Repl • Change FSName replication factor • Choose excess replicas – Rack number do not decrease – Get block from least available disk space node • Add to invalidateSets(to-be-deleted block set) • ReplicationMonitor compute blocks to be deleted for each DN • On next DN’s heartbeat, give delete block command to DN • DN delete specified blocks • Update blocksMap when DN send blockReport 43
  • 44. One DN down • DataNode stop sending heartbeat • NameNode – HeartbeatMonitor find DN dead when doing heartbeat check – Remove all blocks belong to DN – Update neededReplications (block set need one or more replication) – ReplicationMonitor compute block to be replicated for each DN – On next DN’s heartbeat, NameNode send replication block command • DataNode – Replicate block 44
  • 46. High Availability • NameNode SPOF – NameNode hold all the meta – If NN crash, the whole cluster unavailable • Though fsimage can recover from SNN – It’s not a up-to-date fsimage • Need HA solutions 46
  • 47. HA Solutions • DRBD • Avatar Node • Backup Node 47
  • 48. HA - DRBD • DRBD (http://www.drbd.org) – block devices designed as a building block to form high availability (HA) clusters. – Like network based raid-1 • Use DRBD to backup NN’s fsimage & editlogs – A cold backup for NN – Restart NN cost no more than 10 minutes 48
  • 49. HA - DRBD • Mirror one of NN’s name dir to a remote node – All name dir is the same • When NN fail – Copy mirrored name dir to all name dir – Restart NN – All will be done in no more than 20 mins 49
  • 50. HA Solutions • DRBD • Avatar Node • Backup Node 50
  • 51. HA - AvatarNode • Complete Hot Standby – NFS for storage of fsimage and editlogs – Standby node Consumes transactions from editlogs on NFS continuously – DataNodes send message to both primary and standby node • Fast Switchover – Less than a minute 51
  • 52. HA - AvatarNode • Active-Standby Pair Client – Coordinated via ZooKeeper – Failover in few seconds Client retrieves block location from Primary – Wrapper over NameNode or Standby • Active AvatarNode Active Write transaction Read transaction Standby AvatarNode – Writes transaction log to NFS AvatarNode (NameNode) NFS (NameNode) filer Filer • Standby AvatarNode Block Block – Reads/Consumes Location Location transactions from NFS filer messages messages – Processes all messages from DataNodes DataNodes – Latest metadata in memory 52
  • 53. HA - AvatarNode • Four steps to failover – Wipe ZooKeeper entry. Clients will know the failover is in progress. (0 seconds) – Stop the primary NameNode. Last bits of data will be flushed to Transaction Log and it will die. (Seconds) – Switch Standby to Primary. It will consume the rest of the Transaction log and get out of SafeMode ready to serve traffic. (Seconds) – Update the entry in ZooKeeper. All the clients waiting for failover will pick up the new connection (0 seconds) • After: Start the first node in the Standby Mode – Takes a while, but the cluster is up and running 53
  • 55. HA Solutions • DRBD • Avatar Node • Backup Node 55
  • 56. HA - BackupNode • NN synchronously streams Client transaction log to Client retrieves block location from BackupNode NN • BackupNode applies log NN Synchronous stream transacton logs to to in-memory and disk (NameNode) BN image BN • BN always commit to disk Block Location (BackupNode) messages before success to NN • If BN restarts, it has to catch up with NN DataNodes 56
  • 57. Tools • More Tools … – Balancer – Fsck – Distcp 57
  • 58. Tools - Balancer • Need Re-Balancing – When new node is add to cluster • bin/start-balancer.sh – Move block from over-utilized node to under-utilized node • dfs.balance.bandwidthPerSec – Control the impact on business • -t <threshold> – Default 10% – stop when difference from average utilization is less than threshold 58
  • 59. Tools - Fsck • hadoop fsck /path/to/file • Check HDFS’s healthy – Missing blocks, corrupt blocks, mis-replicated blocks … • Get blocks & locations of files – hadoop fsck /path/to/file -files -blocks -locations 59
  • 60. Tools - Distcp • Inter-cluster copy – hadoop distcp -i –pp -log /logdir hdfs://srcip/srcpath/ /destpath – Use map-reduce(actually maps) to start a distributed-fashion copy • Also fast copy in the same cluster 60
  • 62. Hadoop Future • Short-circuit local reads – dfs.client.read.shortcircuit = true – Available in hadoop-1.x or cdh3u4 • Native checksums (HDFS-2080) • BlockReader keepalive to DN (HDFS-941) • “Zero-copy read” support (HDFS-3051) • NN HA (HDFS-3042) • HDFS Federation • HDFS RAID 62
  • 63. References • Tom White, Hadoop The definitive guide • http://hadoop.apache.org/hdfs/ • Hadoop WiKi – HDFS – http://wiki.apache.org/hadoop/HDFS • Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung, The Google File System – http://research.google.com/archive/gfs.html • Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler , The Hadoop Distributed File System – http://storageconference.org/2010/Papers/MSST/Shvachko.pdf 63
  • 64. The End Thank You Very Much! chiangbing@gmail.com 64