Mapr Mirroring details (Steps from CLDB logs )
A mirror volume is a read-only physical copy of another volume, i.e source volume. We can use mirror volumes in the same cluster (local mirroring) to provide local load balancing by using mirror volumes to serve read requests for the most frequently accessed data in the cluster or as backup solution incase someone drops/deletes source volume.
This Blog assumes you have 5.2 secure cluster with correct ticket for access and permission to initiate mirror. Below are command line steps to create mirror / Same can be achieved via MCS .
1) Create source volume.
1) Create source volume.
maprcli volume create -name srcvol -path /srcvol
2) Create mirror volume "mirrorvol" which is mirror of source volume "srcvol" .
maprcli volume create -name mirrorvol -source srcvol@Source -path /mirrorvol -type mirror
3) Loading source volume with some data for us to mirror it gradually.
dd if=/dev/zero of=11gfile bs=65536 count=163840
hadoop fs -put 11gfile /srcvol/
4) Now start the mirror.
[mapr@node9 logs]$ maprcli volume mirror start -name mirrorvol
messages
Started mirror operation for volume(s) 'mirrorvol'
[mapr@node9 logs]$
In this blog we will review CLDB logs to understand detailed steps/Stages during mirroring.
i) Mirroring starts (STATE_MIRROR_STARTED ):
2017-03-27 15:50:31,869 INFO VolumeUpdateManager [RPC-2]: VolumeUpdate: VolName: mirrorvolStarting mirror op STATE_UPDATE
ii) Now collects details of source and mirror volume as a setup process :
2017-03-27 15:50:31,894 INFO VolumeMirror [RPC-2]: Starting mirror of volume mirrorvol, cluster Source volumeId 19731699, mirrortype MIRROR_TYPE_LIVE, rollforward post mirror true, snap expiry time 0, dest creds uid 2000, src creds uid 2000 binding 43470384 cldbPort 7222 restoreSessionId 0
2017-03-27 15:50:31,951 INFO VolumeMirrorInfo [pool-5-thread-1]: Read mirror status of volume mirrorvol is STATE_MIRROR_STARTED
2017-03-27 15:50:32,160 INFO VolumeMirrorInfo [pool-5-thread-3]: Got the list of containers in the destination volumemirrorvol count 1 number of master nodes for volume 1
iii) Create mirrorsnapshot on source .
2017-03-27 15:50:33,458 INFO VolumeMirrorInfo [pool-5-thread-5]: Creating source snapshot mirrorsrcsnap.19731699.27-Mar-2017-15-50-33 of volume srcvol@Source src vol id 2238200 src root cid 2243 for mirroring of mirrorvol@Sourceenabled featurelist - mfs.feature.devicefile.support, mfs.feature.tables, mfs.feature.db.spillv2.support, mfs.feature.db.ace.support, mfs.feature.db.regionmerge.support, mfs.feature.filecipherbit.support, mfs.feature.db.bulkload.support, mfs.feature.db.repl.support, mfs.feature.db.json.support, mfs.feature.rwmirror.support, mfs.feature.fileace.support, mfs.feature.hardlinks.support, mfs.feature.volume.upgrade, mfs.feature.audit.support
2017-03-27 15:50:33,863 INFO VolumeMirrorInfo [pool-5-thread-5]: Created Source volume snapshotmirrorsrcsnap.19731699.27-Mar-2017-15-50-33 snapshotId 256000049 for src volume srcvol@Source mirror volume mirrorvol@Source dataSrcSnapCreateTimeMillis 1490655033490
iv) Gets list if containers on source Snapshot for mirroring .
2017-03-27 15:50:34,018 INFO VolumeMirrorInfo [pool-5-thread-6]: Got the list of containers in the source snapshot for mirorring of mirrorvol@Source src snapshot name mirrorsrcsnap.19731699.27-Mar-2017-15-50-33 count 6
v) Create mirrosnapshot on destination.
2017-03-27 15:50:35,304 INFO VolumeMirrorInfo [pool-5-thread-8]: Creating destination snapshot mirrorsnap.27-Mar-2017-15-50-35 of volume mirrorvol@Source dest vol id 19731699 dest rw cid 2244 snap expiry time 1491259835303
2017-03-27 15:50:35,418 INFO VolumeMirrorInfo [pool-5-thread-8]: Created destination volume snapshotmirrorsnap.27-Mar-2017-15-50-35 snapshotId 256000050 for src volume srcvol@Source mirror volume mirrorvol@Source retry count 0
vi) Now CLDB role is only to get update on status for mirror progress from MFS ( MFS are directly involved in resync 1-1 ) . Status changes to STATE_MIRROR_RESYNC_INPROGRESS .
2017-03-27 15:50:35,959 INFO VolumeUpdateManager [RPC-6]: Saving the src vol snapshot id 256000049 src vol snapshot name mirrorsrcsnap.19731699.27-Mar-2017-15-50-33
2017-03-27 15:50:35,985 INFO VolumeMirrorInfo [pool-5-thread-10]: Update mirror status of volume mirrorvol@Source newstate STATE_MIRROR_RESYNC_INPROGRESS mirrorid 1 nextMirrorId 2 completed successfully. srcsnapshot name mirrorsrcsnap.19731699.27-Mar-2017-15-50-33 destsnap name mirrorsnap.27-Mar-2017-15-50-35 destsnap id 256000050 srcsnap id 256000049
During this resync process if containers are needed for Mapping containers from source to mirror volume they are created.
2017-03-27 15:50:36,135 INFO ContainerAllocator [RPC-2]: ContainerCreate Container ID:2254 vol:19731699 Servers: 10.10.70.109
2017-03-27 15:50:36,136 INFO ContainerAllocator [RPC-9]: ContainerCreate Container ID:2252 vol:19731699 Servers: 10.10.70.109
2017-03-27 15:50:36,136 INFO ContainerAllocator [RPC-4]: ContainerCreate Container ID:2251 vol:19731699 Servers: 10.10.70.109
2017-03-27 15:50:36,137 INFO ContainerAllocator [RPC-7]: ContainerCreate Container ID:2253 vol:19731699 Servers: 10.10.70.109
2017-03-27 15:50:36,137 INFO ContainerAllocator [RPC-8]: ContainerCreate Container ID:2250 vol:19731699 Servers: 10.10.70.109
v) Resync Completes .
2017-03-27 15:53:34,922 INFO VolumeMirrorInfo [VolumeMirrorThread0]: Completed resync of containers of volume mirrorvol@Source
vi) Now the Status of the mirror changes to STATE_MIRROR_ROLLFORWARD_INPROGRESS. This stage is where snapshot containers in destination mirror volume are promoted to latest Copy which is part of Read only mirrored volume.
2017-03-27 15:53:35,067 INFO VolumeMirrorInfo [pool-5-thread-5]: Update mirror status of volume mirrorvol@Source newstate STATE_MIRROR_ROLLFORWARD_INPROGRESS mirrorid 1 nextMirrorId 2 completed successfully. srcsnapshot name mirrorsrcsnap.19731699.27-Mar-2017-15-50-33 destsnap name mirrorsnap.27-Mar-2017-15-50-35 destsnap id 256000050 srcsnap id 256000049
2017-03-27 15:53:35,301 INFO VolumeMirrorInfo [VolumeMirrorThread0]: Completed rollforward of containers of volume mirrorvol@Source
vii) Now source mirror snapshot is removed.
2017-03-27 15:53:35,352 INFO VolumeMirrorInfo [pool-5-thread-3]: Removing source snapshot id 256000049 of volume srcvol@Source for mirroring of mirrorvol@Source op StartMirror OpFailed false
2017-03-27 15:53:35,360 INFO SnapshotProcessor [RPC-2]: SnapshotRemove: Removing snapshot mirrorsrcsnap.19731699.27-Mar-2017-15-50-33 with snapshotid 256000049 for volume srcvol
2017-03-27 15:53:35,417 INFO PurgeExecutor [pool-1-thread-2]: [purgeSnapshot] Deleted snapshot with Id: 256000049
viii) Finally State for mirror changes to complete (STATE_MIRROR_COMPLETE)
2017-03-27 15:53:35,476 INFO VolumeMirrorInfo [pool-5-thread-9]: Update mirror status of volume mirrorvol@Source newstate STATE_MIRROR_COMPLETE mirrorid 1 nextMirrorId 2 completed successfully. srcsnapshot name destsnap name mirrorsnap.27-Mar-2017-15-50-35 destsnap id 256000050 srcsnap id 0
2017-03-27 15:53:35,552 INFO VolumeMirrorInfo [VolumeMirrorThread0]: Mirroring successfully completed for volume mirrorvol@Source From srcvol@Source
Some commands of interest :
To know 1:1 mapping between containers of mirror and source volume we can run below command to get the details.
[mapr@node9 logs]$ maprcli dump volumeinfo -volumename mirrorvol -json |egrep "ContainerId|CreatorContainerId|NameContainer"
"ContainerId":2244,
"NameContainer":"true",
"CreatorContainerId":2243,
"ContainerId":2250,
"NameContainer":"false",
"CreatorContainerId":2248,
"ContainerId":2251,
"NameContainer":"false",
"CreatorContainerId":2245,
"ContainerId":2252,
"NameContainer":"false",
"CreatorContainerId":2246,
"ContainerId":2253,
"NameContainer":"false",
"CreatorContainerId":2247,
"ContainerId":2254,
"NameContainer":"false",
"CreatorContainerId":2249,
Below command gives details on % of mirror completed (mirror-percent-complete) and volume type ("volumetype":1 indicates mirror volume) .
[mapr@node9 logs]$ maprcli volume info -name mirrorvol -json
{
"timestamp":1490660616571,
"timeofday":"2017-03-27 05:23:36.571 GMT-0700",
"status":"OK",
"total":1,
"data":[
{
"acl":{
"Principal":"User mapr",
"Allowed actions":[
"dump",
"restore",
"m",
"a",
"d",
"fc"
]
},
"creator":"mapr",
"aename":"mapr",
"aetype":0,
"numreplicas":"3",
"minreplicas":"2",
"nsNumReplicas":"3",
"nsMinReplicas":"2",
"allowGrant":"false",
"reReplTimeOutSec":"0",
"replicationtype":"high_throughput",
"rackpath":"/data",
"mirrorthrottle":"1",
"accesstime":"March 27, 2017",
"readonly":"1",
"mountdir":"/mirrorvol",
"volumename":"mirrorvol",
"mounted":1,
"quota":"0",
"advisoryquota":"0",
"snapshotcount":"1",
"logicalUsed":"10255",
"used":"1300",
"snapshotused":"0",
"totalused":"1300",
"scheduleid":0,
"schedulename":"",
"mirrorscheduleid":0,
"volumetype":1,
"mirrortype":2,
"creatorcontainerid":2243,
"creatorvolumeuuid":"-5484106972208618544:-9062173038321204647",
"volumeid":19731699,
"actualreplication":[
0,
100,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"nameContainerSizeMB":0,
"nameContainerId":2244,
"needsGfsck":false,
"maxinodesalarmthreshold":"0",
"dbrepllagsecalarmthresh":"0",
"limitspread":"true",
"partlyOutOfTopology":0,
"auditVolume":0,
"audited":0,
"coalesceInterval":60,
"enableddataauditoperations":"getattr,setattr,chown,chperm,chgrp,getxattr,listxattr,setxattr,removexattr,read,write,create,delete,mkdir,readdir,rmdir,createsym,lookup,rename,createdev,truncate,tablecfcreate,tablecfdelete,tablecfmodify,tablecfScan,tableget,tableput,tablescan,tablecreate,tableinfo,tablemodify,getperm,getpathforfid,hardlink",
"disableddataauditoperations":"",
"mirrorSrcVolume":"srcvol",
"mirrorSrcVolumeId":2238200,
"mirrorSrcCluster":"Source",
"mirrorDataSrcVolume":"srcvol",
"mirrorDataSrcVolumeId":2238200,
"mirrorDataSrcCluster":"Source",
"lastSuccessfulMirrorTime":1490655215453,
"mirror-percent-complete":100,
"mirrorId":2,
"nextMirrorId":2,
"mirrorstatus":0,
"volumeAces":{
"readAce":"p",
"writeAce":"p"
},
"fixCreatorId":"false",
"ReplTypeConversionInProgress":"0"
}
]
}
No comments:
Post a Comment