Turning on FileClient Debug dynamically

                  Turning on FileClient Debug dynamically 


    As explained in earlier blogs Enabling FC debug can be very helpful while debugging issues and seeing what calls the client is making to the cluster incase there are timeout's , retry errors but no obvious issues at the cluster side. While the below blogs explain way to turn on FC debug when the process starts up it can be priceless if FC can be turned on dynamically.

http://abizeradenwala.blogspot.com/2016/11/interacting-with-hadoop-cluster-file.html
http://abizeradenwala.blogspot.com/2016/11/file-client-debug-logging.html

Lets say we need to get FC debug logs for NM pid .

1) Get NM pid .

[root@node107rhel72 logs]# jps | grep 12531

12531 NodeManager

[root@node107rhel72 logs]# 

2) Run below command to list the details on shared memory segments. From below commands we know the shmid of interest is 1006731277 .

[root@node107rhel72 logs]# ipcs -mp | grep 12531

------ Shared Memory Creator/Last-op PIDs --------

shmid      owner      cpid       lpid    

1006731277 mapr       12531      17541     

1006764046 mapr       12531      12531     

1006796815 mapr       12531      12531     

[root@node107rhel72 logs]# ipcs -m | egrep "bytes|1234"

------ Shared Memory Segments --------

key        shmid      owner      perms      bytes      nattch     status 

0x00000000 0          mapr       644        1234       0                       

0x00000000 1002995714 mapr       644        1234       1          dest               

0x00000000 1006731277 mapr       644        1234       1          dest         

[root@node107rhel72 logs]# 

3) Now since we know the Shmid we can enable FC DEBUG via below command .

/opt/mapr/server/tools/fcdebug -s 1006731277 -l DEBUG

4) Verify debug messages are logged .

i) Std out of the process will log messages as below.

[root@node107rhel72 logs]# tailf yarn-mapr-nodemanager-node107rhel72.out

2018-03-20 19:48:42,1632 DEBUG Client fs/client/fileclient/cc/client.cc:3686 Thread: 19570 atime in millisec =  1520406282000

2018-03-20 19:48:42,1632 DEBUG Client fs/client/fileclient/cc/client.cc:3703 Thread: 19570 nlink - 1

2018-03-20 19:48:42,1632 DEBUG Client fs/client/fileclient/cc/client.cc:3526 Thread: 19570 PathWalk: Adding /var/mapr/local/node107rhel72/mapred/nodeManager/fidservers Fid 2113.38.1969838 to Fidcache

2018-03-20 19:48:42,1632 DEBUG Client fs/client/fileclient/cc/client.cc:3530 Thread: 19570 PathWalk: WalkDone File /var/mapr/local/node107rhel72/mapred/nodeManager/fidservers, resp fid 2113.38.1969838

2018-03-20 19:48:42,1632 DEBUG JniCommon fs/client/fileclient/cc/jni_MapRClient.cc:1147 Thread: 19570  -- Exit JNI getattr -- /var/mapr/local/node107rhel72/mapred/nodeManager/fidservers

                            or

ii) You can also verify if debugs are turned on for the module in question.

[root@node107rhel72 logs]# /opt/mapr/server/tools/fcdebug -i -s 1006731277

              Global :      DEBUG 

                 RPC :      DEBUG 

       MapServerFile :      DEBUG 

        MapServerDir :      DEBUG 

           Container :      DEBUG 

            Snapshot :      DEBUG 

                Util :      DEBUG 

         Replication :      DEBUG 

           PunchHole :      DEBUG 

             KvStore :      DEBUG 

            Truncate :      DEBUG 

           Orphanage :      DEBUG 

          FileServer :      DEBUG 

           Heartbeat :      DEBUG 

               Defer :      DEBUG 

       ServerCommand :      DEBUG 

               Write :      DEBUG 

                  DB :      DEBUG 

               DBRpc :      DEBUG 

         HighLatency :      DEBUG 

             DBLocks :      DEBUG 

          DBMemIndex :      DEBUG 

        DBPermission :      DEBUG 

              DBRepl :      DEBUG 

            AceCache :      DEBUG 

              Marlin :      DEBUG 

            MarlinLG :      DEBUG 

           LogStream :      DEBUG 

          FileAccess :      DEBUG 

           FSProfile :      DEBUG 

                NFSD :      DEBUG 

            Cidcache :      DEBUG 

              Client :      DEBUG 

            Fidcache :      DEBUG

5) Turn of  FC debug dynamically after getting all the logs of interest .

[root@node107rhel72 logs]# /opt/mapr/server/tools/fcdebug -s 1006731277

Applying default level on all modules 

[root@node107rhel72 logs]#

Note:-  If finding shmid is challenge you can run below command against the PID of interest and Debug logging will be enabled on the fly after a minute.

[root@node107rhel72 logs]# for shm in `ipcs -mp | grep -w 12531 | cut -f 1 -d " "`; do echo $shm ;/opt/mapr/server/tools/fcdebug  -s $shm -l DEBUG; done

1006731277

1006764046

1006796815

[root@node107rhel72 logs]#