CLDB guts

                                                     CLDB Guts

We need a high level metric that tells us the cluster state, cldbguts utility monitors the activity of cluster.  Monitoring the progress may be useful when troubleshooting cluster issues or estimate time cluster would take during cluster restart and make sure progress is made in right direction. Running this command from the CLDB master node will provide most relevant information. 

/opt/mapr/bin/cldbguts containers

2017-03-22 20:28:46,208

             assigns                              roles                                      progress                                                              con-chain         

      nr       nc    nrt   nct      bm       ms         wr  rs     vr           uu      m%        uc               v%       tr                ms          1r            2r   

      1        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      7        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      9        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      1        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      4        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      3        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      1        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      1        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

      2        0      0     0     207053  2663695   2   2  5659657  120    92.78%  2870869  97.57%  8530529    2663695  2864682  2794964 

Container-Chain : Below 3 are most direct stats to review and confirm if CLDB is making progress in bringing containers online i.e every ticking second if the counters are increasing means progress is made in right direction .

ms: Number of unique containers that have a master.

1r: Number of unique containers that have 2 valid copies of the data

2r: Number of unique containers that have 3 valid copies of the data

Assigns : 

nr: Number of container assign requests in the 1 second

nc: Number of containers created as part of the container assign requests in the 1 second

nrt: Number of container assign requests for tablets in the past 1 second

nct: Number of containers created as part of the above container assign requests for tablets in past 1 second

roles :

rs: Number of replica containers that are resyncing.

vr: Number of non-master replica containers that have finished resynchronization

progress :

m%: Percentage of containers that have valid masters
uc: Number of unique containers
v%: Percentage of replica containers that are valid (i.e., have completed resynchronization)
tr: Total number of replica containers.

/opt/mapr/bin/cldbguts heartbeat

2017-03-22 17:44:21,241

         mfs hb                           nfs hb          

      nr          pt               to        nr           pt      

     551        194604      0       564        3863399    

     539        173896      0       568        3719581    

     552        169888      0       560        3734112    

     554        177049      0       575        3786113    

     551        177405      0       571        3830968    

     561        181276      0       578        3871850    

     557        181475      0       560        3876045    

     562        170063      0       568        3664293    

     560        188553      0       576        3750587    

nr: Number of MapR-FS/NFS heartbeats completed in the past 1 second. 

to: Number of MapR-FS/NFS heartbeats that took longer than expected in the past 1 second.

 /opt/mapr/bin/cldbguts rpc

2017-03-22 17:45:15,760

                acr                           fcr                    regn        

      nr         pt          to  d  dp       nr       pt     to      nr      pt    to 

     185        145      0  0   0       0         0      0      0       0      0 

     204        207      0  0   0       4        339     0      0       0      0 

     165        108      0  0   0       0         0      0      0       0      0 

     175        153      0  0   0       0         0      0      0       0      0 

     198        351      0  0   0       0         0      0      0       0      0 

     172        396      0  0   0       4        277     0      0       0      0 

     187        158      0  0   0       0         0      0      0       0      0 

     193        156      0  0   0       0         0      0      0       0      0 

     165        118      0  0   0       7        589     0      0       0      0 

nr: Number of ACRs/FCR/Registration completed in the past 1 second.
to: Number of ACRs/FCR/Registration that took longer than expected in the past 1 second.

/opt/mapr/bin/cldbguts alarms

2017-03-22 17:45:34,953

         nodes        

   nn   of       sp   of 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

  594  112  1800  56 

nn/sp: Number of nodes/SP
of: Offline nodes/SP

/opt/mapr/bin/cldbguts table

2017-03-22 17:45:57,140

            location                         size                       sptable         

      lu        up     dl  sc       lu         up     dl  sc    lu    up       dl    sc 

      0          0      0   0        3        519      0   0     0     0       0     -1 

      0          3      0   0        1        501      0   0     0     0       1      0 

      0          0      0   0        0        457      0   0     0     0       0      0 

      0          0      0   0        0        560      0   0     0     0       0      0 

      0          1      0   0       10        500      0   0     0     0       1      0 

     512         8      0   0        0        498      0   0     0     0       8      0 

      0          1      0   0       27        543      0   0     0     0       1      0 

      0          1      0   0       28        643      0   0     0     0       0      0 

      0          0      0   0        0        440      0   0     0     0       0      0 

Iu/up/dl: inserts/lookups/deletes by table 
sc: Number of active scans by table

MapR-DB Table Replication on Secure cluster( Locally )

                                MapR-DB Table Replication on Secure cluster( Locally )


This is another flavor of my prior blog where i need to have table replication locally.
http://abizeradenwala.blogspot.com/2016/03/mapr-db-table-replication.html

1) Make sure cluster knows about the gateway.

maprcli cluster gateway set -dstcluster <des_cluster_name> -gateways <dest_gateway_hostname> 

--Confirm cluster can resolve the G/W

[root@node10 ~]# maprcli cluster gateway resolve -dstcluster Destination

Source              GatewayHosts       

mapr-clusters.conf  10.10.70.110:7660

2) Create and load the source table.

[root@node10 ~]# echo "create '/user/mapr/AbiLocalt1', {NAME => 'c1', VERSIONS => 1}" | hbase shell

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

create '/user/mapr/AbiLocalt1', {NAME => 'c1', VERSIONS => 1}

0 row(s) in 0.9310 seconds

Hbase::Table - /user/mapr/AbiLocalt1

[root@node10 ~]# echo "put '/user/mapr/AbiLocalt1', 'r1', 'c1', '$(date)'" | hbase shell 

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

put '/user/mapr/AbiLocalt1', 'r1', 'c1', 'Thu Mar 23 13:01:51 PDT 2017'

0 row(s) in 0.0850 seconds

3) Now setup replica for the source table.

maprcli table replica autosetup -path /user/mapr/AbiLocalt1 -replica /user/mapr/AbiLocalreplt1

4) Now scan the destination table to check if table data is replicated.

[root@node10 ~]# echo "scan '/user/mapr/AbiLocalreplt1'" | hbase shell 

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

scan '/user/mapr/AbiLocalreplt1'

ROW                                           COLUMN+CELL                                                                                                                          

 r1                                           column=c1:, timestamp=1490299319331, value=Thu Mar 23 13:01:51 PDT 2017                                                              

1 row(s) in 0.1080 seconds

5) Finally to test the replication I will push 2 rows and confirm replica stream writes to destination table.

[root@node10 ~]# echo "put '/user/mapr/AbiLocalt1', 'r1', 'c1', 'abi'" | hbase shell 

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

put '/user/mapr/AbiLocalt1', 'r1', 'c1', 'abi'

0 row(s) in 0.0860 seconds

[root@node10 ~]# echo "put '/user/mapr/AbiLocalt1', 'r2', 'c1', 'abizer'" | hbase shell 

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

put '/user/mapr/AbiLocalt1', 'r2', 'c1', 'abizer'

0 row(s) in 0.1070 seconds

Source table :- 

[root@node10 ~]# echo "scan '/user/mapr/AbiLocalt1'" | hbase shell 

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

scan '/user/mapr/AbiLocalt1'

ROW                                           COLUMN+CELL                                                                                                                          

 r1                                           column=c1:, timestamp=1490299674775, value=abi                                                                                       

 r2                                           column=c1:, timestamp=1490299746021, value=abizer                                                                                    

2 row(s) in 0.1110 seconds

Destination table :- 

[root@node10 ~]# echo "scan '/user/mapr/AbiLocalreplt1'" | hbase shell 

HBase Shell; enter 'help<RETURN>' for list of supported commands.

Type "exit<RETURN>" to leave the HBase Shell

Version 0.98.12-mapr-1605, r1b36a7c15cda3f507e22826b9062ccb778af694c, Fri Jun  3 04:45:12 UTC 2016

Not all HBase shell commands are applicable to MapR tables.

Consult MapR documentation for the list of supported commands.

scan '/user/mapr/AbiLocalreplt1'

ROW                                           COLUMN+CELL                                                                                                                          

 r1                                           column=c1:, timestamp=1490299674775, value=abi                                                                                       

 r2                                           column=c1:, timestamp=1490299746021, value=abizer                                                                                    

2 row(s) in 0.1080 seconds

Check Gateway logs to make sure replication stream is pushing the data :-

2017-03-23 13:09:06,0250 INFO Replicator fs/gateway/src/cc/replicator.cc:552 Thread: 0 FamilyMap Name c1 famId src: 1 dst: 1

2017-03-23 13:09:08,1536 INFO Replicator fs/gateway/src/cc/replicator.cc:1799 Thread: 0 >FlushStream id: 8636429065916677938 bucket 2063.39.131436 maxSrcBucketOff 0 flushOff 53 numRows 1 atime 1490299748153 KeyRange[r2, r2]

2017-03-23 13:09:08,1564 INFO Replicator fs/gateway/src/cc/replicator.cc:1813 Thread: 0 <FlushStream id: 8636429065916677938 bucket 2063.39.131436 startOff 0 maxOff 53 TimeRange[1490299746021, 1490299746021] table /user/mapr/AbiLocalreplt1

Understanding MapR DB Binary Table

                                                  Understanding MapR DB  Binary Table

• A Table - an infinitely large key/value space composed of tablets ( In above diagram table spans from 1-100 key range ). Table data is organized into rows and columns and columns are organized into column families ( CF's ).
  
  
  
  
  
• A Tablet/Region -  4-6GB of data that represents a particular key range. Table comprises of many tablets ( Tablet 1 {key 1~50} and Tablet 2 {key 50~100} ).
  
  
  
  
  
• Partition - It is a mini tablet of up to about 2 GB (partition split in two at 2GB) with zero or one active buckets ( Tablet 1 => Partition 1 [Key 1~25] + Partition 2 [Key 25~50])
• an active bucket is accepting puts
• a second bucket might be flushing (or not exist)
  
  
• Bucket -  It is a file that holds incoming writes unordered up to about 150MB. The keys are kept in in-memory index. If we run low on cache space this will force a bucket flush.
  
  
• Segment - Collection of read only spill files that typically holds 2-4MB of sorted data (just like HBase HFiles) with the row key index at the start of the file
• all of the values for a given row are stored together
• all of the versions for the values are also stored together
  

  

• Spill file - a file that contains sorted additions (called spills) to the corresponding segment. The first spill file can be thought of as a “the initial segment.”
• a segment has no more than 6 spills
• We consider the segment to be composed of the spills so really the first spill is the initial segment data.

  
  

  From Segment we get Spill Map:

  [root@node9 ~]# maprcli debugdb  dump -fid 2182.34.131316 -json

  {

  "timestamp":1489941805408,

  "timeofday":"2017-03-19 09:43:25.408 GMT-0700",

  "status":"OK",

  "total":1,

  "data":[

  {

  "key":"",

  "value":{

  "fid":"<parentCID>.35.131318"

  }

  }

  ]

  }

  
  

  From Spill Map we get Spill (keys)

  
  

  [root@node9 ~]# maprcli debugdb  dump -fid 2182.35.131318 -json

  {

  "timestamp":1489941819033,

  "timeofday":"2017-03-19 09:43:39.033 GMT-0700",

  "status":"OK",

  "total":3,

  "data":[

  {

  "key":0,

  "numRemoteBlocks":0,

  "numSpills":0,

  "numSegments":0,

  "value":{

  "fid":"<parentCID>.36.131320",

  "smeSize":51,

  "keyIdxOffset":12,

  "keyIdxLength":57,

  "ldbIdxLength":20,

  "bloomBitsPerKey":8000,

  "numLogicalBlocks":2,

  "numPhysicalBlocks":5,

  "numRows":1,

  "numRowsWithDelete":0,

  "families":{

  "id":1,

  "offset":524288,

  "length":43,

  "minTimeStamp":1489471080975,

  "maxTimeStamp":1489471080975

  }

  }

  },

  {

  "key":1,

  "numRemoteBlocks":0,

  "numSpills":0,

  "numSegments":0,

  "value":{

  "fid":"<parentCID>.38.131324",

  "smeSize":51,

  "keyIdxOffset":12,

  "keyIdxLength":57,

  "ldbIdxLength":20,

  "bloomBitsPerKey":8000,

  "numLogicalBlocks":2,

  "numPhysicalBlocks":5,

  "numRows":1,

  "numRowsWithDelete":0,

  "families":{

  "id":1,

  "offset":524288,

  "length":43,

  "minTimeStamp":1489471253234,

  "maxTimeStamp":1489471253234

  }

  }

  },

  {

  "key":2,

  "numRemoteBlocks":0,

  "numSpills":0,

  "numSegments":0,

  "value":{

  "fid":"<parentCID>.38.131324",

  "smeSize":52,

  "keyIdxOffset":8204,

  "keyIdxLength":57,

  "ldbIdxLength":20,

  "bloomBitsPerKey":8000,

  "numLogicalBlocks":2,

  "numPhysicalBlocks":5,

  "numRows":1,

  "numRowsWithDelete":0,

  "families":{

  "id":1,

  "offset":589824,

  "length":43,

  "minTimeStamp":1489471657238,

  "maxTimeStamp":1489471657238

  }

  }

  }

  ]

  }