Fuse installation and DEBUG

                                                 Fuse Installation and DEBUG

This post assumes you have MapR cluster already setup and need to setup Fuse client for reading and writing to cluster.

Mapr posix package needs maps client package installed on the node as it has dependency on it.

[root@node10 logs]# repoquery --requires mapr-posix-client-basic

/bin/sh

mapr-client

[root@node10 logs]# 

1)  Install Basic POSIX client package on your machine.

yum install mapr-posix-client-basic


2)  Run configure.sh to set this node as the client node.

For example:

/opt/mapr/server/configure.sh -N <clustername> -C <CLDBhost> -Z <ZooKeeperhost> -c -R

3) Start maps-posix-client-basic .

service mapr-posix-client-basic start
           or

/opt/mapr/initscripts/mapr-posix-client-basic start

Once Up you will see below logs in /opt/mapr/logs/posix-client-basic.log.

Sun Oct 16 00:35:26 PDT 2016: Running /opt/mapr/initscripts/mapr-posix-client-basic start

Sun Oct 16 00:35:26 PDT 2016: Mounting posix-client-basic /mapr --log_path /opt/mapr/logs -o allow_other -o big_writes -o auto_unmount -o async_dio -o max_background=64 at /mapr ...

Starting fuse with 1 libraries

Sun Oct 16 00:35:26 PDT 2016: Result:0

Started registering to cldb.Successfully registered to cldb.Invalid Option, record already found, cannot override secure=true for cluster: Abizer.cluster.com

Below FS will be mounted from which the client will have access to the cluster.

[root@node10 logs]# df -hP /mapr

Filesystem          Size  Used Avail Use% Mounted on

posix-client-basic  168G  437M  168G   1% /mapr

[root@node10 logs]# 

Troubleshooting :

1) You can force unmount via below command if needed. 

umount -lf /mapr

2) If fuse is hanging 

i) Run gstack <pid> (of the fuse process) in a loop with a sleep of 1 second in the state where fio makes no progress to check the state of fuse process for say 5 mins.

ii)  It would be helpful if we enable debug logs. We can use fcdebug to enable debug logs at run-time ( Say for 5 mins ).

Find Shmid (1262092291) this is logged in /opt/mapr/logs/ffs.log-0 logs when fuse starts up.

2016-10-16 00:35:26,7650 ERROR FuseAPI fs/client/fileclient/cc/fuse_api.cc:1357 Thread: 11050 Shmid to be used by fcdebug 1262092291

Enable Debug for all modules will take 60 seconds for debug logs to print .

 /opt/mapr/server/tools/fcdebug -s 1262092291 -l DEBUG

After you enable Debug and any operation via fuse client will log messages as below.

2016-10-17 22:47:13,6229 DEBUG FuseOps fs/client/fuse/cc/fuse_ops_ll.c:597 Thread: 11123 lookup: parent 550024249360, name abi, libindex 0



2016-10-17 22:47:13,6229 DEBUG Client fs/client/fileclient/cc/client.cc:3119 Thread: 11123 Lookupfid : start = abi, end = (nil)



2016-10-17 22:47:13,6230 DEBUG Client fs/client/fileclient/cc/client.cc:3220 Thread: 11123 Pathwalk of abi started, pfid 2049.16.2



2016-10-17 22:47:13,6230 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:729 Thread: 11123 GetBinding: for cid 2049 isUpd 0



2016-10-17 22:47:13,6230 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:563 Thread: 11123 >allocate and fill for cid 2049, isUpd 0



2016-10-17 22:47:13,6230 DEBUG Cidcache fs/client/fileclient/cc/cidcache.cc:517 Thread: 11123 Found valid entry for cid 2049, numSer 1



2016-10-17 22:47:13,6230 DEBUG Client fs/client/fileclient/cc/client.cc:3223 Thread: 11123 Rpc: PathWalkPlusProc to Fileserver: 10.10.70.110:5692 for fid: 2049.16.2



2016-10-17 22:47:13,6237 DEBUG ClntProfileRpc fs/client/fileclient/cc/client.cc:3223 Thread: 11123 Profile: CltRpcDone: server 10.10.70.110:5692 took 0 msec error 2 FID 2049.16.2  PathWalkPlus path abi

iii)  You may have to run "ulimit -c unlimited" and then "kill -6 <fuse-pid>" or "kill -11 <Fuse-pid> to get core for the process.

This details would be good enough along with application logs to know the timestamp when customer observed delays to start troubleshooting.