Run Pig Jobs with Oozie          

This Blog assumes below Oozie and Pig package are already installed on running MapR cluster and Steps from blog 1 is already followed.

mapr-oozie-4.1.0.201606271017-1.noarch
mapr-oozie-internal-4.1.0.201606271017-1.noarch
mapr-pig-0.14.201608040131-1.noarch

http://abizeradenwala.blogspot.com/2015/07/installing-oozie-and-running-sample-job.html

Since Oozie is current bundled with Pig v0.12 we will need below steps for oozie pig action to work.

/opt/mapr/oozie/oozie-4.1.0/share1/lib/pig/pig-withouthadoop-0.12.1-mapr-1408-h2.jar
/opt/mapr/oozie/oozie-4.1.0/share1/lib/pig-2/pig-withouthadoop-0.12.1-mapr-1408-h2.jar

1) The Oozie share/lib directory has two sets of JAR files for Pig.  We will use the Pig JAR files from the share/lib/pig-2 directory with MapR distribution versions 4.0.0 and later.

To specify the JAR files for a given Pig job, add the following section to the workflow.xml file:

<name>oozie.action.sharelib.for.pig</name>

<value>pig-2</value>

2) Stop Oozie:

maprcli node services -name oozie -action stop -nodes <nodes>

3)  Remove all files located within the /opt/mapr/oozie/oozie<version>/share2/lib/pig*/ directory EXCEPT the oozie-sharelib-pig-<version>-mapr.jar file.
Now copy new Pig jars to share lib location,

cp <PIG_HOME>/pig-core-h2.jar <OOZIE_HOME>/share2/lib/pig-2/

cp <PIG_HOME>/lib/* <OOZIE_HOME>/share2/lib/pig-2/ 

4) Remove the zookeeper jars .

rm -rf <OOZIE_HOME>/share2/lib/pig-2/zookeeper*.jar

5) Now move all the old jars in latest share lib in MaprFS to temp location.

hadoop fs -mv /oozie/share/lib/lib_20160804181903/pig-2/* /abizer

And now copy latest jars into share lib in MaprFS

hadoop fs -put /opt/mapr/oozie/oozie-4.1.0//share2/lib/pig-2/*   /oozie/share/lib/lib_20160804181903/pig-2

6) Copy work-flow.xml to maprfs which is specified in job.properties file

hadoop fs -put workflow.xml /user/mapr/examples/apps/pig/workflow.xml

Example of my workflow.xml
[mapr@node3 pig-2]$ cat /opt/mapr/oozie/oozie-4.1.0/examples/apps/pig/workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.2" name="pig-wf">
    <start to="pig-node"/>
    <action name="pig-node">
        <pig>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/output-data/pig"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
                <property>
                        <name>oozie.action.sharelib.for.pig</name>
                        <value>pig-2</value>
                 </property>
            </configuration>
            <script>id.pig</script>
            <param>INPUT=/user/${wf:user()}/input-data/text</param>
            <param>OUTPUT=/user/${wf:user()}/output-data/pig</param>
        </pig>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Pig failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

7 ) Start Oozie:

maprcli node services -name oozie -action start -nodes <nodes>

8) As user MapR i am running sample workflow.

[mapr@node3 root]$ /opt/mapr/oozie/oozie-4.1.0/bin/oozie job -oozie="http://localhost:11000/oozie" -config /opt/mapr/oozie/oozie-4.1.0/examples/apps/pig/job.properties -run
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/mapr/oozie/oozie-4.1.0/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
job: 0000000-160805145748028-oozie-mapr-W

9) On checking the status pig wf was successfully executed by Oozie.

[mapr@node3 root]$ /opt/mapr/oozie/oozie-4.1.0/bin/oozie job -info 0000000-160805145748028-oozie-mapr-W -oozie="http://localhost:11000/oozie"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/mapr/oozie/oozie-4.1.0/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
Job ID : 0000000-160805145748028-oozie-mapr-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : pig-wf
App Path      : maprfs:/user/mapr/examples/apps/pig
Status        : SUCCEEDED
Run           : 0
User          : mapr
Group         : -
Created       : 2016-08-05 18:58 GMT
Started       : 2016-08-05 18:58 GMT
Last Modified : 2016-08-05 18:59 GMT
Ended         : 2016-08-05 18:59 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000000-160805145748028-oozie-mapr-W@:start:                                  OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------
0000000-160805145748028-oozie-mapr-W@pig-node                                 OK        job_1470423487588_0001 SUCCEEDED  -        
------------------------------------------------------------------------------------------------------------------------------------
0000000-160805145748028-oozie-mapr-W@end                                      OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------

[mapr@node3 root]$

Rename SP in MapR

                                                 Rename SP in MapR

Sometimes for consistency Admin would want SP name to be same as the old SP name even after SP is recreated. Usually after SP is recreated it doesn't have the same original SP name .


Below are the steps to rename an SP. 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp list 
ListSPs resp: status 0:2 
No. of SPs (2), totalsize 584687 MB, totalfree 583859 MB 

SP 0: name SP1, Online, size 290295 MB, free 289881 MB, path /dev/sdb 
SP 1: name SP2, Online, size 294391 MB, free 293978 MB, path /dev/sde 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp offline /dev/sde 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp list 
ListSPs resp: status 0:2 
No. of SPs (2), totalsize 290295 MB, totalfree 289881 MB 

SP 0: name SP1, Online, size 290295 MB, free 289881 MB, path /dev/sdb 
SP 1: name SP2, Offline, size 307197 MB, free 0 MB, path /dev/sde 

[root@qa-node-111 ~]# /opt/mapr/server/mrconfig sp rename -n SP4 /dev/sde 
SP renamed to SP4 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp online /dev/sde 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp list 
ListSPs resp: status 0:2 
No. of SPs (2), totalsize 584687 MB, totalfree 583859 MB 

SP 0: name SP1, Online, size 290295 MB, free 289881 MB, path /dev/sdb 
SP 1: name SP4, Online, size 294391 MB, free 293977 MB, path /dev/sde 

Now SP2 is rename to SP4 as needed. 

Configuring the Fair Scheduler with ACL on MapR Cluster

                   Configuring the Fair Scheduler with ACL on MapR Cluster

This blog assumes you have MapR 4.0.2 un-secure cluster installed.

1) Add below lines  to yarn-site.xml on all Resource Managers nodes followed by restart the RM.

vi /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/yarn-site.xml

<property><name>yarn.admin.acl</name><value>mapr</value></property>

<property><name>yarn.acl.enable</name><value>true</value></property>

With this setting MapR is administrator for yarn cluster and can kill any job and submit jobs no matter what. By default, yarn.admin.acl is set to *, which means anyone can be the Admin.

Note: An empty value for the yarn.admin.acl is not considered a valid value by YARN and it will fall back on the value configured in the yarn-default.xml which will allow access to allow everyone

2) Now add below settings to fair-scheduler.xml

vi /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/fair-scheduler.xml

<allocations>

<queue name="root">

<aclSubmitApps>mapr</aclSubmitApps>                                       <aclAdministerApps>mapr</aclAdministerApps>

<queue name="mapr">

<minResources>20000 mb,40 vcores,5 disks</minResources>

<maxResources>30000 mb,50 vcores,50 disks</maxResources>

<maxRunningApps>10</maxRunningApps>

<weight>1.0</weight>

<schedulingPolicy>fair</schedulingPolicy>

<aclSubmitApps>mapr</aclSubmitApps>

</queue>

<queue name="abizer">

<minResources>20000 mb,40 vcores,5 disks</minResources>

<maxResources>30000 mb,50 vcores,50 disks</maxResources>

<maxRunningApps>10</maxRunningApps>

<weight>1.0</weight>

<schedulingPolicy>fair</schedulingPolicy>

<aclSubmitApps>abizer</aclSubmitApps>

</queue>

</queue>

</allocations>

3)  Below output shows Queue ACL for each user where :

i) root user has no permission to submit or Administer any queue

[root@master hadoop]# hadoop queue -showacls

Queue acls for user :  root

Queue  Operations

=====================

root 

root.abizer 

root.default 

root.mapr 

Eg 1 [root@master ~]# yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar pi 1 2

Number of Maps  = 1

Samples per Map = 2

Wrote input for Map #0

Starting Job

16/04/14 21:42:28 INFO input.FileInputFormat: Total input paths to process : 1

16/04/14 21:42:28 INFO mapreduce.JobSubmitter: number of splits:1

16/04/14 21:42:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460696891161_0009

16/04/14 21:42:28 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager

16/04/14 21:42:28 INFO impl.YarnClientImpl: Submitted application application_1460696891161_0009

16/04/14 21:42:28 INFO mapreduce.JobSubmitter: Cleaning up the staging area maprfs:/var/mapr/cluster/yarn/rm/staging/root/.staging/job_1460696891161_0009

java.io.IOException: Failed to run job : User root cannot submit applications to queue root.root

            at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:321)

Eg 2 [root@master ~]# yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar pi -Dmapreduce.job.queuename=root.abizer 1 2

Number of Maps  = 1

Samples per Map = 2

Wrote input for Map #0

Starting Job

16/04/14 21:42:13 INFO input.FileInputFormat: Total input paths to process : 1

16/04/14 21:42:13 INFO mapreduce.JobSubmitter: number of splits:1

16/04/14 21:42:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460696891161_0008

16/04/14 21:42:14 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager

16/04/14 21:42:14 INFO impl.YarnClientImpl: Submitted application application_1460696891161_0008

16/04/14 21:42:14 INFO mapreduce.JobSubmitter: Cleaning up the staging area maprfs:/var/mapr/cluster/yarn/rm/staging/root/.staging/job_1460696891161_0008

java.io.IOException: Failed to run job : User root cannot submit applications to queue root.abizer

ii) Mapr user had permission to Administer any queue and can submit app to any queue

[mapr@master hadoop]$ hadoop queue -showacls

Queue acls for user :  mapr

Queue  Operations

=====================

root  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
root.abizer  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
root.mapr  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
root.root  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS

[mapr@master root]$ yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar pi -Dmapreduce.job.queuename=root.abizer 1 2

Number of Maps  = 1

Samples per Map = 2

Wrote input for Map #0

Starting Job

16/04/14 21:40:34 INFO input.FileInputFormat: Total input paths to process : 1

16/04/14 21:40:34 INFO mapreduce.JobSubmitter: number of splits:1

16/04/14 21:40:34 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460696891161_0007

16/04/14 21:40:35 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager

16/04/14 21:40:35 INFO impl.YarnClientImpl: Submitted application application_1460696891161_0007

16/04/14 21:40:35 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1460696891161_0007/

16/04/14 21:40:35 INFO mapreduce.Job: Running job: job_1460696891161_0007

16/04/14 21:40:43 INFO mapreduce.Job: Job job_1460696891161_0007 running in uber mode : false

16/04/14 21:40:43 INFO mapreduce.Job:  map 0% reduce 0%

16/04/14 21:40:49 INFO mapreduce.Job:  map 100% reduce 0%

16/04/14 21:40:55 INFO mapreduce.Job:  map 100% reduce 100%

16/04/14 21:40:55 INFO mapreduce.Job: Job job_1460696891161_0007 completed successfully

16/04/14 21:40:55 INFO mapreduce.Job: Counters: 46

            File System Counters

                        FILE: Number of bytes read=0

                        FILE: Number of bytes written=162489

                        FILE: Number of read operations=0

                        FILE: Number of large read operations=0

                        FILE: Number of write operations=0

                        MAPRFS: Number of bytes read=336

                        MAPRFS: Number of bytes written=303

                        MAPRFS: Number of read operations=43

                        MAPRFS: Number of large read operations=0

                        MAPRFS: Number of write operations=59

            Job Counters

                        Launched map tasks=1

                        Launched reduce tasks=1

                        Data-local map tasks=1

                        Total time spent by all maps in occupied slots (ms)=4040

                        Total time spent by all reduces in occupied slots (ms)=10986

                        Total time spent by all map tasks (ms)=4040

                        Total time spent by all reduce tasks (ms)=3662

                        Total vcore-seconds taken by all map tasks=4040

                        Total vcore-seconds taken by all reduce tasks=3662

                        Total megabyte-seconds taken by all map tasks=4136960

                        Total megabyte-seconds taken by all reduce tasks=11249664

                        DISK_MILLIS_MAPS=2020

                        DISK_MILLIS_REDUCES=4870

            Map-Reduce Framework

                        Map input records=1

                        Map output records=2

                        Map output bytes=18

                        Map output materialized bytes=0

                        Input split bytes=134

                        Combine input records=0

                        Combine output records=0

                        Reduce input groups=2

                        Reduce shuffle bytes=24

                        Reduce input records=2

                        Reduce output records=0

                        Spilled Records=4

                        Shuffled Maps =1

                        Failed Shuffles=0

                        Merged Map outputs=2

                        GC time elapsed (ms)=40

                        CPU time spent (ms)=1000

                        Physical memory (bytes) snapshot=780922880

                        Virtual memory (bytes) snapshot=5446422528

                        Total committed heap usage (bytes)=722468864

            Shuffle Errors

                        IO_ERROR=0

            File Input Format Counters

                        Bytes Read=118

            File Output Format Counters

                        Bytes Written=97

Job Finished in 21.633 seconds

Estimated value of Pi is 4.00000000000000000000

iii) Abizer user had permission to submit app to “root.abizer“ queue and cannot administer any queue except itself (default behavior)

[abizer@master hadoop]$ hadoop queue -showacls

Queue acls for user :  abizer

Queue  Operations

=====================

root 

root.abizer  SUBMIT_APPLICATIONS

root.default 

root.mapr 

[abizer@master root]$ yarn jar /opt/mapr/hadoop/hadoop-2.5.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1-mapr-1501.jar pi -Dmapreduce.job.queuename=root.mapr 1 2

Number of Maps  = 1

Samples per Map = 2

Wrote input for Map #0

Starting Job

16/04/14 21:38:29 INFO input.FileInputFormat: Total input paths to process : 1

16/04/14 21:38:29 INFO mapreduce.JobSubmitter: number of splits:1

16/04/14 21:38:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460696891161_0006

16/04/14 21:38:30 INFO security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager

16/04/14 21:38:30 INFO impl.YarnClientImpl: Submitted application application_1460696891161_0006

16/04/14 21:38:30 INFO mapreduce.JobSubmitter: Cleaning up the staging area maprfs:/var/mapr/cluster/yarn/rm/staging/abizer/.staging/job_1460696891161_0006

java.io.IOException: Failed to run job : User abizer cannot submit applications to queue root.mapr

4) Audit logging :

The Resource Manager will log attempts to kill or move applications. These messages are logged by the RMAuditLogger as part of standard operational logging. 

For Eg : Mapr started application application_1460696891161_0018 and when user Abizer tries to kill the application you will see log message as below "User doesn't have permissions to MODIFY_APP"

2016-04-14 22:16:55,433 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=mapr     IP=10.10.70.112 OPERATION=Submit Application Request    TARGET=ClientRMService  RESULT=SUCCESS  APPID=application_1460696891161_0018

2016-04-14 22:17:24,489 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=abizer1  IP=10.10.70.112 OPERATION=Kill Application Request      TARGET=ClientRMService  RESULT=FAILURE  DESCRIPTION=Unauthorized user   PERMISSIONS=User doesn't have permissions to MODIFY_APP APPID=application_1460696891161_0018

To conclude Queue level setting (aclSubmitApps, aclAdministerApps) and YANR admin ACL setting (yarn.acl.enable ,yarn.admin.acl) both the setting have to be in place, even if you disable access through one way, if the user has permission through the queue setting or admin then the user will be able to kill other users job or submit job in another users queue.

Note :- Application Master Link might not be accessible in RM UI due to above change to make the link working please change hadoop.http.staticuser.user to mapr user and restart HS . 

To do so add below property to core-site.xml 

<property><name>hadoop.http.staticuser.user</name><value>mapr</value></property> 

MapR-DB Table Replication on Secure Cluster

                                         MapR-DB Table Replication on Secure cluster

                                                   

This Blog assumes you have 2 clusters up and running securely

Note :-

Source Cluster  - Node 10.10.70.111

Destination Cluster  - Node 10.10.70.109  

Steps to setup mapr-db table replication in secure environment are as below. 

1) On all node in SOURCE CLUSTER verify that maprserverticket , cldb.key , ssl_truststore, ssl_keystore are same. Run md5sum on these file on each node to confirm. 
2) On all node in DESTINATION CLUSTER verify that maprserverticket , cldb.key , ssl_truststore, ssl_keystore are same. Run md5sum on these file on each node to confirm. 
3) Copy /opt/mapr/conf/ssl_truststore from DESTINATION CLUSTER to cldb node of SOURCE CLUSTER under /tmp/ and run the below command to merge ssl_truststore on SOURCE CLUSTER. 

Note: Ignore ssl_truststore merge step if in case you have already done it earlier. 
$ chmod 644 /opt/mapr/conf/ssl_truststore 
$ /opt/mapr/server/manageSSLKeys.sh merge /tmp/ssl_truststore /opt/mapr/conf/ssl_truststore 
$ chmod 444 /opt/mapr/conf/ssl_truststore 

4) Copy the merged truststore file '/opt/mapr/conf/ssl_truststore' on all the node in SOURCE CLUSTER under /opt/mapr/conf/ 

5) Generate cross-cluster ticket from DESTINATION CLUSTER , in this case i created ticket to last for 10 years. 

$ maprlogin generateticket -type crosscluster -out /tmp/destination-ticket -duration 3650:0:0 

Note: - It is critical to specify an appropriate value for the duration. After the ticket expires, communication between the clusters will stop. In this example, the duration of ten years is given for convenience of explanation. Use a value that is consistent with your security policies.

6) Copy file /tmp/destination-ticket from DESTINATION CLUSTER to SOURCE CLUSTER's cldb node under /tmp. 

7) At SOURCE CLUSTER append the content of file /tmp/destination-ticket in /opt/mapr/conf/maprserverticket .

$ cat /tmp/destination-ticket >> /opt/mapr/conf/maprserverticket 

8) Copy file /opt/mapr/conf/maprserverticket on all the nodes in SOURCE CLUSTER . 

9) Stop zookeeper and warden in SOURCE CLUSTER followed by starting ZK and then warden once ZK is up

10) On SOURCE CLUSTER create user ticket for user mapr for source and destination cluster .

maprlogin password

maprlogin password -cluster Dest

cat /tmp/maprticket_2000

Source KV34qQ0jtmQXObJglDiZqqHHm507pbYOsHd4qIEEavC+0PGDlB/YeTBGReOxf+EleSEO78pYvNqzoqK5uK+5Gibx0v+XPEyl2UuDgBR6GUBwx4yUUxnUY7Ct4STdcHmvcyE47AVM4gXc9ivQCvkokyIvZwYiGtwVQ8rnTNrLuzuUPAH8GMbR486UgMQ8axy8QIcA2zexIT0K0Ct7Fj612UPVonXZDfnAB2yG5gEhdmxLOMPmQLm9qt6f49Pzrn96IwHGLXQtUAmfrTwrbPPPOSUshA==

Dest 4D9Z469Y3j7h3sy2CVZwQrlXDEWHCtmCENQQGFvVzoGsytXp4K3OLOf+BZhLIoTBZuu2uzmV/1SbnqYUfO9NXsxAx3Bomez9iZ3ni7Kfk9m9CTEPydl9updp8IFQZ83jQ7IERM3WgN/rouEg3T/BnwPA2+U2cnGjeeCgXH3lmopJGiYFCegXWhhn9TmKawH0Vp4f3tDBBo2nWjr1sCnBvsBXhYP6DQzA3vLdmbGWQn6d2IJRNUA0irG8MSjxzZ4E9y4S2hu4gnLYE0IXgXNoWWhawQ==

11) Create table and put some data. 

$ echo "create '/user/mapr/AbiSourcet1', {NAME => 'c1', VERSIONS => 1}" | hbase shell 
$ echo "put '/user/mapr/AbiSourcet1', 'r1', 'c1', '$(date)'" | hbase shell 

12)  Install mapr-gateway package at DESTINATION CLUSTER if not present and make SOURCE SIDE aware of G/W. 

On DESTINATION CLUSTER : yum install mapr-gayeway -y   ( Configure.sh and restart of cluster will be needed for warden to manage this service )

On Source Cluster : maprcli cluster gateway resolve -dstcluster Dest  ( Command to verify source cluster 

Source              GatewayHosts                                                         can resolve Destination G/W )

mapr-clusters.conf  10.10.70.109:7660 

Command to make Source cluster aware of destination G/W

On Source Cluster: $ maprcli cluster gateway set -dstcluster <des_cluster_name> -gateways <dest_gateway_hostname> 

13)  Now set up and start replication between a source table and replica table.

$ maprcli table replica autosetup -path /mapr/Source/user/mapr/AbiSourcet1 -replica /mapr/Dest/user/mapr/AbiDestt1 

Above command does below steps in background :

1. Creates a table on the replication cluster with the required column families
2. Declares the new table to be a replica of the source table with a paused replication state.
3. Declares the source table as an upstream source for the replica.
4. Runs the CopyTables utility to load a copy of the source data into the replica.
5. Clears the paused replication state to start the replication stream.

14) Add new row in table from source side and verify from destination side if upgrade was received on replica to make sure replication is working as expected 

echo "put '/user/mapr/AbiSourcet1', 'r4', 'c1', '$(date)'" | hbase shell