hadoop增加机架感知

1.新增脚本rack-topology.sh,rack_topology.data在到 /etc/hadoop/conf下

rack-topology.sh
---------------


#!/bin/bash

# Adjust/Add the property "net.topology.script.file.name"
# to core-site.xml with the "absolute" path the this
# file. ENSURE the file is "executable".

# Supply appropriate rack prefix
RACK_PREFIX=default

# To test, supply a hostname as script input:
if [ $# -gt 0 ]; then

CTL_FILE=${CTL_FILE:-"rack_topology.data"}

HADOOP_CONF=${HADOOP_CONF:-"/etc/hadoop/conf"}

if [ ! -f ${HADOOP_CONF}/${CTL_FILE} ]; then
echo -n "/$RACK_PREFIX/rack "
exit 0
fi
while [ $# -gt 0 ] ; do
nodeArg=$1
exec< ${HADOOP_CONF}/${CTL_FILE}
result=""
while read line ; do
ar=( $line )
if [ "${ar[0]}" = "$nodeArg" ] ; then
result="${ar[1]}"
fi
done
shift
if [ -z "$result" ] ; then
echo -n "/$RACK_PREFIX/rack "
else
echo -n "/$RACK_PREFIX/rack_$result "
fi
done

else
echo -n "/$RACK_PREFIX/rack "
fi

rack_topology.data
---------------

127.18.4.60 01
127.18.4.59 02
127.18.4.58 03
127.18.4.57 04

rack-topology.sh必须要有可执行权限

chmod u+x rack-topology.sh

测试是否可行
bash rack-topology.sh 127.18.4.60

若是结果返回 /default/rack_01的话,那么就是脚本ok的。

2.在core-site.xml 中添加一下配置


<property>
<name>net.topology.script.file.name</name>
<value>/etc/hadoop/conf/rack-topology.sh</value>
</property>

3.将配置下放到所有节点中并重启 namenode , datanode
4.确认是否生效
在hadoop的namenode日志下找到
org.apache.hadoop.net.NetworkTopology: Adding a new node

若是有这样的日志的话,那么就是ok了。

或者通过

hdfs dfsadmin -printTopology

来查看是否机架感知是否启用!

参考资料:http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_hdfs-administration/content/ch05.html#create_rack_topology_script

发表评论?

0 条评论。

发表评论


注意 - 你可以用以下 HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>