Label based Scheduling (YARN)

This blog assumes you already have atleast 2 nodes 5.2.1 cluster cluster installed . Below is diagram for the steps we will go through below .

 To set up node labels for the purpose of scheduling YARN applications (including MapReduce applications) on a specific node or group of nodes follow below steps:

       1.   Create a text file and specify the labels you want to use for the nodes in your cluster. In this example, the file is named node.labels with 2 nodes.

1. 
   [mapr@vm52 root]$ cat node.labels
   
   vm52-1 fast
   
   vm52 slow
   
   [mapr@vm52 root]$
   

2. Copy the file to a location on MapR-FS where it will not be modified or deleted, such as /var/mapr.

   hadoop fs -put ~/node.labels /var/mapr

3. Edit yarn-site.xml on all ResourceManager nodes and set the node.labels.file parameter and the optional node.labels.monitor.interval parameter as shown:

4. <property>
      <name>node.labels.file</name>
      <value>/var/mapr/node.labels</value>
      <description>The path to the node labels file.</description>
   </property>
   
   <property>
      <name>node.labels.monitor.interval</name>
      <value>120000</value>
      <description>Interval for checking the labels file for updates (default is 2 min)</description>
   </property>

     

      4.  Modify fair-scheduler.xml to add "fast" label to MapR queue .

1. 
   [mapr@vm52 hadoop]$ cat fair-scheduler.xml 
   
   <allocations>
   
   <queue name="root">
   
    <aclSubmitApps>mapr</aclSubmitApps>                                      
   
    <aclAdministerApps>mapr</aclAdministerApps>
   
   
   <queue name="mapr">
   
   <minResources>20000 mb,1 vcores,0 disks</minResources>
   
   <maxResources>30000 mb,4 vcores,2 disks</maxResources>
   
   <maxRunningApps>10</maxRunningApps>
   
   <weight>1.0</weight>
   
   <label>fast</label>
   
   <schedulingPolicy>fair</schedulingPolicy>
   
   <aclSubmitApps>mapr</aclSubmitApps>
   
   </queue>
   
   <queue name="abizer">
   
   <minResources>20000 mb,40 vcores,5 disks</minResources>
   
   <maxResources>30000 mb,50 vcores,50 disks</maxResources>
   
   <maxRunningApps>10</maxRunningApps>
   
   <weight>1.0</weight>
   
   <schedulingPolicy>fair</schedulingPolicy>
   
   <aclSubmitApps>abizer</aclSubmitApps>
   
   </queue>
   
   
   
   
   
   
   
   </queue>
   
   
   
   
   </allocations>
   
   [mapr@vm52 hadoop]$
   

     5.   Restart ResourceManager to set up the labels from the node labels file for the first time. For subsequent   changes to take effect, issue following commands to manually tell the ResourceManager to reload the node labels file .

• For any YARN applications, including MapReduce jobs, enter "yarn rmadmin -refreshLabels" to refresh labels if any changes are done.

  6. Verify that labels are implemented correctly by running following commands:   

1. 
   [mapr@vm52 hadoop]$ yarn rmadmin -showLabels
   
   17/10/04 00:33:58 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to vm52/10.10.70.99:8033
   
                     Nodes     Labels
   
                    vm52-1     [fast]
   
                      vm52     [slow]
   
   [mapr@vm52 hadoop]$ 
   

   7.      Running a sample teragen Job in mapr queue for it to run on fast label nodes only .

1. 
   [mapr@vm52 root]$ hadoop jar /opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0-mapr-1607.jar teragen -Dmapreduce.job.queue=mapr 100000000 /teragen21
   
   17/10/04 00:58:58 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to vm52/10.10.70.99:8032
   
   17/10/04 00:59:00 INFO terasort.TeraSort: Generating 100000000 using 2
   
   17/10/04 00:59:00 INFO mapreduce.JobSubmitter: number of splits:2
   
   17/10/04 00:59:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1506992471684_0015
   
   17/10/04 00:59:03 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager
   
   17/10/04 00:59:03 INFO impl.YarnClientImpl: Submitted application application_1506992471684_0015
   
   17/10/04 00:59:03 INFO mapreduce.Job: The url to track the job: http://vm52:8088/proxy/application_1506992471684_0015/
   
   17/10/04 00:59:03 INFO mapreduce.Job: Running job: job_1506992471684_0015
   
   17/10/04 00:59:12 INFO mapreduce.Job: Job job_1506992471684_0015 running in uber mode : false
   
   17/10/04 00:59:12 INFO mapreduce.Job:  map 0% reduce 0%
   
   17/10/04 00:59:23 INFO mapreduce.Job:  map 33% reduce 0%
   
   17/10/04 01:07:30 INFO mapreduce.Job:  map 66% reduce 0%
   
   17/10/04 01:07:31 INFO mapreduce.Job:  map 100% reduce 0%
   
   17/10/04 01:07:32 INFO mapreduce.Job: Job job_1506992471684_0015 completed successfully
   
   17/10/04 01:07:32 INFO mapreduce.Job: Counters: 33
   
     File System Counters
   
     FILE: Number of bytes read=0
   
     FILE: Number of bytes written=192826
   
     FILE: Number of read operations=0
   
     FILE: Number of large read operations=0
   
     FILE: Number of write operations=0
   
     MAPRFS: Number of bytes read=170
   
     MAPRFS: Number of bytes written=10000000000
   
     MAPRFS: Number of read operations=20
   
     MAPRFS: Number of large read operations=0
   
     MAPRFS: Number of write operations=201171874
   
    Job Counters 
   
     Killed map tasks=1
   
     Launched map tasks=2
   
     Other local map tasks=2
   
     Total time spent by all maps in occupied slots (ms)=495359
   
     Total time spent by all reduces in occupied slots (ms)=0
   
     Total time spent by all map tasks (ms)=495359
   
     Total vcore-seconds taken by all map tasks=495359
   
     Total megabyte-seconds taken by all map tasks=507247616
   
     DISK_MILLIS_MAPS=247680
   
    Map-Reduce Framework
   
     Map input records=100000000
   
     Map output records=100000000
   
     Input split bytes=170
   
     Spilled Records=0
   
     Failed Shuffles=0
   
     Merged Map outputs=0
   
     GC time elapsed (ms)=961
   
     CPU time spent (ms)=195000
   
     Physical memory (bytes) snapshot=474079232
   
     Virtual memory (bytes) snapshot=3744894976
   
     Total committed heap usage (bytes)=317718528
   
    org.apache.hadoop.examples.terasort.TeraGen$Counters
   
     CHECKSUM=214760662691937609
   
    File Input Format Counters 
   
     Bytes Read=0
   
    File Output Format Counters 
   
     Bytes Written=10000000000
   
   [mapr@vm52 root]$ 
   

VERIFY :-

  We can see containers are only running of fast node (vm52-1)

1. 
   [mapr@vm52 hadoop]$  yarn node -list
   
   17/10/04 01:00:30 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to vm52/10.10.70.99:8032
   
   Total Nodes:2         
   
   Node-Id      Node-State Node-Http-Address Number-of-Running-Containers      
   
   vm52:56287         RUNNING         vm52:8042                            0    
   
   vm52-1:35535         RUNNING       vm52-1:8042                            2
   
   [mapr@vm52 hadoop]$