MapR-DB Tablet consistency Check

                        MapR-DB Tablet consistency Check

This post assumes you have /abizer table created and its empty for us to now use below utility to load the table.

1)  /opt/mapr/server/tools/loadtest -mode put  -numfamilies 1 -numcols 8 -numrows 115450  -table /abizer

23:44:31    0 secs     47391 rows    47391 rows/s    0ms latency   74ms maxLatency

23:44:32    1 secs     69275 rows    21884 rows/s    0ms latency   62ms maxLatency

23:44:33    2 secs     85854 rows    16579 rows/s    1ms latency  332ms maxLatency

23:44:34    3 secs     91451 rows     5597 rows/s    3ms latency  326ms maxLatency

23:44:35    4 secs     99718 rows     8267 rows/s    2ms latency  370ms maxLatency

23:44:36    5 secs    104277 rows     4559 rows/s    4ms latency  665ms maxLatency

23:44:37    6 secs    106115 rows     1838 rows/s   11ms latency  161ms maxLatency

23:44:38    7 secs    115450 rows     9335 rows/s    5ms latency  343ms maxLatency

Overall Rate 14425.84 rows/s, Latency 1ms

2) Now check the size and other attributes of table 

 maprcli table info -path /abizer -json

{

"timestamp":1481356005057,

"timeofday":"2016-12-09 11:46:45.057 GMT-0800",

"status":"OK",

"total":4,

"data":[

{

"path":"/abizer",

"numregions":4,

"totallogicalsize":866525184,

"totalphysicalsize":538484736,

"totalcopypendingsize":261562368,

"totalrows":173733,

"totalnumberofspills":522,

"totalnumberofsegments":219,

"autosplit":true,

"bulkload":false,

"insertionorder":true,

"tabletype":"binary",

"regionsizemb":4096,

"audit":false,

"maxvalueszinmemindex":100,

"adminaccessperm":"u:root",

"createrenamefamilyperm":"u:root",

"bulkloadperm":"u:root",

"packperm":"u:root",

"deletefamilyperm":"u:root",

"replperm":"u:root",

"splitmergeperm":"u:root",

"defaultappendperm":"u:root",

"defaultcompressionperm":"u:root",

"defaultmemoryperm":"u:root",

"defaultreadperm":"u:root",

"defaultversionperm":"u:root",

"defaultwriteperm":"u:root",

"uuid":"315a20f5-ad00-3638-3229-03afb04b5800"

}

]

}

3) Now since we know there are 4 regions lets check details of 4 regions (Regions also called Tablet) in the table with respective details.

maprcli table region list -path /abizer -json

{

"timestamp":1481356069293,

"timeofday":"2016-12-09 11:47:49.293 GMT-0800",

"status":"OK",

"total":4,

"data":[

{

"primarymfs":"node10.maprlab.local:5660",

"secondarymfs":"",

"startkey":"-INFINITY",

"endkey":"user3051486366437214287",

"lastheartbeat":0,

"fid":"2248.32.131418",

"logicalsize":215515136,

"physicalsize":133914624,

"copypendingsize":127434752,

"numberofrows":42943,

"numberofrowswithdelete":0,

"numberofspills":147,

"numberofsegments":52

},

{

"primarymfs":"node10.maprlab.local:5660",

"secondarymfs":"",

"startkey":"user3051486366437214287",

"endkey":"user5105987298073629634",

"lastheartbeat":0,

"fid":"2247.32.131310",

"logicalsize":216645632,

"physicalsize":134545408,

"copypendingsize":0,

"numberofrows":43089,

"numberofrowswithdelete":0,

"numberofspills":149,

"numberofsegments":53

},

{

"primarymfs":"node10.maprlab.local:5660",

"secondarymfs":"",

"startkey":"user5105987298073629634",

"endkey":"user720538676648464684",

"lastheartbeat":0,

"fid":"2251.32.131262",

"logicalsize":217849856,

"physicalsize":135897088,

"copypendingsize":0,

"numberofrows":44198,

"numberofrowswithdelete":0,

"numberofspills":112,

"numberofsegments":56

},

{

"primarymfs":"node10.maprlab.local:5660",

"secondarymfs":"",

"startkey":"user720538676648464684",

"endkey":"INFINITY",

"lastheartbeat":0,

"fid":"2250.32.131186",

"logicalsize":216514560,

"physicalsize":134127616,

"copypendingsize":134127616,

"numberofrows":43503,

"numberofrowswithdelete":0,

"numberofspills":114,

"numberofsegments":58

}

]

}

4) To get Info/details of local tablets per node alternatively can also be viewed by mrconfig commands.

/opt/mapr/server/mrconfig dbinfo tablets

----------------------

|From Instance 5660::|

----------------------

tablet 2250.32.131186 nref 0 npartitions 1 logicalMB 206 physicalMB 127 rows 43503 splitState None attrAutoSplit 1 tabletSplitThreshSizeMB 6144 partitionSplitThreshSizeMB 2048 isReadOnly 0 error 0 updateError 0

tablet 2251.32.131262 nref 0 npartitions 1 logicalMB 207 physicalMB 129 rows 44198 splitState None attrAutoSplit 1 tabletSplitThreshSizeMB 6144 partitionSplitThreshSizeMB 2048 isReadOnly 0 error 0 updateError 0

tablet 2247.32.131310 nref 0 npartitions 2 logicalMB 206 physicalMB 128 rows 43089 splitState None attrAutoSplit 1 tabletSplitThreshSizeMB 6144 partitionSplitThreshSizeMB 2048 isReadOnly 0 error 0 updateError 0

tablet 2248.32.131418 nref 0 npartitions 2 logicalMB 205 physicalMB 127 rows 42943 splitState None attrAutoSplit 1 tabletSplitThreshSizeMB 6144 partitionSplitThreshSizeMB 2048 isReadOnly 0 error 0 updateError 0

5) Lets say we believe "2251.32.131262" fid has some issues and we want to review the stats of the fid in question, below command can give the info needed.

[root@node10 ~]# maprcli debugdb statTablet -fid 2251.32.131262 -json

{

"timestamp":1481356575116,

"timeofday":"2016-12-09 11:56:15.116 GMT-0800",

"status":"OK",

"total":1,

"data":[

{

"numPhysicalBlocks":16589,

"numLogicalBlocks":26593,

"numRows":44198,

"numRowsWithDelete":0,

"numRemoteBlocks":0,

"numSpills":112,

"numSegments":56

}

]

}

6)  To check the consistency of the tablet we can run below command which check and reports details related to tablet. The startkey and endkey needs to be from output received from "maprcli table region list -path /abizer -json" for the region/Tablet in question.

 If this fails for any reason we need to check /opt/mapr/logs/mfs.log-5 on the master node for the tablet and see if we see the error reported to debug further but normally if there are no issues with the Tablet we would see clean output as below.

[root@node10 ~]# maprcli debugdb checkTablet -fid 2251.32.131262 -tracefile /tmp/tablet-2251.32.131262 -startkey 'user5105987298073629634' -endkey 'user720538676648464684'

TabletRangeCheck done

[root@node10 ~]# cat /tmp/tablet-2251.32.131262 

Tablet 2251.32.131262

Segmap 2251.607.132412

spill 2251.266.131730 parent 2251.265.131728 idx 0 size 2048000

keymap 2251.266.131730 parent 2251.265.131728 idx 0 offset 12 len 31471

cf 2251.266.131730 parent 2251.265.131728 idx 0 family 1 offset 524288 len 2110718

cf 2251.266.131730 parent 2251.265.131728 idx 0 family 2 offset 2686976 len 1048712

Spillmap 2251.265.131728 parent 2251.607.132412

spill 2251.390.131978 parent 2251.383.131964 idx 0 size 2088960

keymap 2251.390.131978 parent 2251.383.131964 idx 0 offset 12 len 32489

cf 2251.390.131978 parent 2251.383.131964 idx 0 family 1 offset 524288 len 2185207

cf 2251.390.131978 parent 2251.383.131964 idx 0 family 2 offset 2752512 len 1050456

Spillmap 2251.383.131964 parent 2251.607.132412

spill 2251.546.132290 parent 2251.383.131964 idx 1 size 1474560

keymap 2251.546.132290 parent 2251.383.131964 idx 1 offset 12 len 19507

cf 2251.546.132290 parent 2251.383.131964 idx 1 family 1 offset 524288 len 1138369

cf 2251.546.132290 parent 2251.383.131964 idx 1 family 2 offset 1703936 len 1138798

[root@node10 ~]# 

Above file proves everything is fine with the tablet and there are no inconsistencies found.