heartBeat : dfs.heartbeat.interval 默认是3秒
blockreport:dfs.blockreport.intervalMsec 默认是6小时,第一次延迟时间dfs.blockreport.initialDelay 默认是0
cachereport:dfs.cachereport.intervalMsec 默认是10秒
触发是:org.apache.hadoop.hdfs.server.datanode.BPServiceActor#offerService 中
基本是按照时间间隔来判断是否需要上报一次cache
lifeline:dfs.datanode.lifeline.interval.seconds 默认是3*dfs.heartbeat.interval
badreport: 在写数据时候,发现由坏块,就上报 org.apache.hadoop.hdfs.server.datanode.DataNode#reportRemoteBadBlock
IBR report : 增量数据上报间隔
dfs.blockreport.incremental.intervalMsec 默认是0 具体是在 org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager 中管理的
heartBeat / blockReport / cacheReport 都会带回namenode中都命令
对应的处理类都在 org.apache.hadoop.hdfs.server.datanode.BPServiceActor#offerService 中
heartBeat
long startProcessCommands = monotonicNow(); if (!processCommand(resp.getCommands()))//处理命令 continue;
blockReport
if ((fullBlockReportLeaseId != 0) || forceFullBr) { cmds = blockReport(fullBlockReportLeaseId); fullBlockReportLeaseId = 0; } processCommand(cmds == null ? null : cmds.toArray(new DatanodeCommand[cmds.size()]));//处理命令
cacheReport
if (!dn.areCacheReportsDisabledForTests()) { DatanodeCommand cmd = cacheReport(); processCommand(new DatanodeCommand[]{ cmd });//处理带回来的命令 }
badreport
public void reportRemoteBadBlock(DatanodeInfo srcDataNode, ExtendedBlock block) throws IOException { BPOfferService bpos = getBPOSForBlock(block); bpos.reportRemoteBadBlock(srcDataNode, block); }
参数设置后的日志
hadoop-hdfs-datanode-jssz-bigdata-datanode-200.log.9:2019-03-31 22:35:51,047 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode jssz-bigdata-namenode-01/10.69.8.34:9010 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000; lifelineIntervalMs=9000
0 条评论。