需要对于yarn集群进行实时和离线作业的分离，就单独搞一个yarn，实时对于资源的竞争比较敏感，所以用了标签来做机器上的隔离来保证资源的隔离性
修改配置：yarn-site.xml
<property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value> </property> <property> <name>yarn.node-labels.manager-class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager</value> </property> <property> <name>yarn.node-labels.enabled</name> <value>true</value> </property> <!-- 这个是hdfs目录，启动用户必须有读写权限 本案是yarn启动--> <property> <name>yarn.node-labels.fs-store.root-dir</name> <value>/user/yarn/node-label</value> </property>
新增capacity-scheduler.xml 在hadoop_conf 配置项下
代码中默认是按照这个名字找的
</div> <div> <configuration> <!-- 定义的队列 --> <property> <name>yarn.scheduler.capacity.root.queues</name> <value>ai_queue,sycpb_queue,common_queue</value> </property> <!-- 队列依赖的资源百分比 --> <property> <name>yarn.scheduler.capacity.root.ai_queue.capacity</name> <value>30</value> </property> <property> <name>yarn.scheduler.capacity.root.sycpb_queue.capacity</name> <value>30</value> </property> <property> <name>yarn.scheduler.capacity.root.common_queue.capacity</name> <value>40</value> </property> <!-- 队列依赖的资源百分比-最大值 --> <property> <name>yarn.scheduler.capacity.root.ai_queue.maximum-capacity</name> <value>45</value> </property> <property> <name>yarn.scheduler.capacity.root.sycpb_queue.maximum-capacity</name> <value>45</value> </property> <property> <name>yarn.scheduler.capacity.root.common_queue.maximum-capacity</name> <value>55</value> </property> <!-- 队列匹配的标签，这个配置很重要 --> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels</name> <value>*</value> </property> <property> <name>yarn.scheduler.capacity.root.ai_queue.accessible-node-labels</name> <value>ai_label</value> </property> <property> <name>yarn.scheduler.capacity.root.sycpb_queue.accessible-node-labels</name> <value>sycpb_label</value> </property> <property> <name>yarn.scheduler.capacity.root.common_queue.accessible-node-labels</name> <value>common_label</value> </property> <!-- root队列下ai_label标签的占用资源比 --> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels.ai_label.capacity</name> <value>30</value> </property> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels.sycpb_label.capacity</name> <value>30</value> </property> <property> <name>yarn.scheduler.capacity.root.accessible-node-labels.common_label.capacity</name> <value>40</value> </property> <!-- root.ai_queue队列下ai_label标签的占用资源比 这个必须是100% --> <property> <name>yarn.scheduler.capacity.root.ai_queue.accessible-node-labels.ai_label.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.sycpb_queue.accessible-node-labels.sycpb_label.capacity</name> <value>100</value> </property> <property> <name>yarn.scheduler.capacity.root.common_queue.accessible-node-labels.common_label.capacity</name> <value>100</value> </property> <!-- root下的默认标签 --> <property> <name>yarn.scheduler.capacity.root.default-node-label-expression</name> <value> ,ai_label,sycpb_label,common_label</value> </property> <!--value为空，则root.default队列只能在没有label的node上执行，而我环境环境中不存在没有label的node，所以提交到root.default的任务卡死--> <property> <name>yarn.scheduler.capacity.root.default.default-node-label-expression</name> <value> </value> </property> <!-- 队列和标签的绑定关系 --> <property> <name>yarn.scheduler.capacity.root.ai_queue.default-node-label-expression</name> <value>ai_label</value> </property> <property> <name>yarn.scheduler.capacity.root.sycpb_queue.default-node-label-expression</name> <value>sycpb_label</value> </property> <property> <name>yarn.scheduler.capacity.root.common_queue.default-node-label-expression</name> <value>common_label</value> </property> </configuration> </div> <div>
常用命令：
添加标签
yarn rmadmin -addToClusterNodeLabels ai_label,sycpb_label,common_label
查询所有标签
yarn cluster –list-node-labels
删除标签
yarn rmadmin -removeFromClusterNodeLabels “ai_label,sycpb_label,common_label”
绑定节点和标签之间的关系
yarn rmadmin -replaceLabelsOnNode jssz-bigdata-datanode-01:8041,ai_label;
查询节点列表
yarn node -list -all
0 条评论。