Script that run's when you install RPM Files

                              Script that run's when you install RPM Files

        

Sometimes we need to find specific file or jars belong to which package while troubleshooting and would like to get same package installed on problematic node as well usually "rpm -qf  <File-name>" works fine. 

Like say i need to find which package in mapr packages mapr fsck utility via below command we get the package name or some jar we see the package where the jar came from.

[root@node1 mapr]# rpm -qf /opt/mapr/server/fsck
mapr-core-internal-4.0.2.29870.GA-1.x86_64

[root@node1 pig-0.14]# rpm -qf pig-0.14.0-mapr-SNAPSHOT.jar
mapr-pig-0.14.201608040131-1.noarch

But now lets take below case where we get pig-core-h2.jar is not owned by any package but we are certain this jar was part of a rpm install yet we don't get rite answer when we query using the file.

[root@node1 pig-0.14]# rpm -qf pig-core-h2.jar 
file /opt/mapr/pig/pig-0.14/pig-core-h2.jar is not owned by any package

This is because rpm packages can run scripts after a package is added or removed . How do we view pre/post install and uninstall rpm file scripts (RHEL)?

The option –scripts to rpm command displays the package specific script(s) that are used as part of the installation and uninstallation processes. Use the following command line option to view rpm file scripts for .rpm file.

rpm -qp --scripts <RPM-Name>.rpm

Example : To list all scripts which will be run as part of "mapr-pig-0.14.201608040131-1.noarch.rpm" install.

[root@node1 pig-0.14]# rpm -qp --scripts /tmp/mapr-pig-0.14.201608040131-1.noarch.rpm 
postinstall scriptlet (using /bin/sh):

# this is install
mkdir -p "/opt/mapr/pig/pig-0.14"/conf
diffR=$(diff -r "/opt/mapr/pig/pig-0.14"/conf "/opt/mapr/pig/pig-0.14"/conf.new | grep "^Only in " | grep "conf.new" | sed "s/^Only in //" | sed "s/: /\//")
for i in `echo $diffR`; do
    j=$(echo $i | sed 's/conf.new/conf/g')
    cp -Rp $i $j 
done

ln -sf /opt/mapr/pig/pig-0.14/bin/pig /opt/mapr/bin/
ln -sf /opt/mapr/pig/pig-0.14/bin/pig /usr/bin/pig


#
# single pig jar for post-yarn releases
#
if [ -f /opt/mapr/MapRBuildVersion ]; then
    MAPR_VERSION=`cat /opt/mapr/MapRBuildVersion | awk -F "." '{print $1"."$2}'`

    #if mapr-core release >=4.0 (yarn beta) returns boolean 1, else returns boolean 0
    POST_YARN=`echo | awk -v cur=$MAPR_VERSION -v min=4.0 '{if (cur >= min) printf("1"); else printf ("0");}'`
    if [[ $POST_YARN == 0 ]]; then
echo POST_YARN=$POST_YARN, removing yarn jars    
        rm -f  /opt/mapr/pig/pig-0.14/pig-core-h1.jar
        mv /opt/mapr/pig/pig-0.14/pig-*-h1.jar /opt/mapr/pig/pig-0.14/pig-core-h1.jar
        mv /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h1.jar /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank.jar
        rm /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h2.jar
        rm /opt/mapr/pig/pig-0.14/pig-*-h2.jar
        mv /opt/mapr/pig/pig-0.14/conf/pig.properties.3.x /opt/mapr/pig/pig-0.14/conf/pig.properties
    else
# mv *-h2.jar to *.jar
echo POST_YARN=$POST_YARN, replacing MR1 jars
 rm -f  /opt/mapr/pig/pig-0.14/pig-core-h2.jar
        mv /opt/mapr/pig/pig-0.14/pig-*-h2.jar /opt/mapr/pig/pig-0.14/pig-core-h2.jar
        mv /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h2.jar /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank.jar
        rm /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h1.jar
        rm /opt/mapr/pig/pig-0.14/pig-*-h1.jar
        rm /opt/mapr/pig/pig-0.14/conf/pig.properties.3.x
    fi    
fi

#
# change ownership
#
DAEMON_CONF="/opt/mapr/conf/daemon.conf"

if [ -f "$DAEMON_CONF" ]; then
    MAPR_USER=$( awk -F = '$1 == "mapr.daemon.user" { print $2 }' $DAEMON_CONF)

    if [ ! -z "$MAPR_USER" ]; then
        chown -R $MAPR_USER /opt/mapr/pig/pig-0.14
    fi
fi

##
## end of post install section
##
preuninstall program: /bin/sh
postuninstall scriptlet (using /bin/sh):

#
# last version is removed
# if package is getting purged remove entire directory
# we may need to remove pig/pig-X/logs
#
if [ "$1" = "0" ]; then
    rm -Rf /opt/mapr/pig/pig-0.14/logs
    rm -Rf /opt/mapr/pig/pig-0.14/conf

    rm -f /opt/mapr/bin/pig
    rm -f /usr/bin/pig
fi

##
## end of post uninstall section
##
[root@node1 pig-0.14]# 

To view scripts for installed packages, enter:

rpm -q --scripts <Package name>

Example :

[root@node1 pig-0.14]# rpm -qa | grep pig
mapr-pig-0.14.201608040131-1.noarch
[root@node1 pig-0.14]# rpm -q --scripts mapr-pig-0.14.201608040131-1.noarch
postinstall scriptlet (using /bin/sh):

# this is install
mkdir -p "/opt/mapr/pig/pig-0.14"/conf
diffR=$(diff -r "/opt/mapr/pig/pig-0.14"/conf "/opt/mapr/pig/pig-0.14"/conf.new | grep "^Only in " | grep "conf.new" | sed "s/^Only in //" | sed "s/: /\//")
for i in `echo $diffR`; do
    j=$(echo $i | sed 's/conf.new/conf/g')
    cp -Rp $i $j 
done

ln -sf /opt/mapr/pig/pig-0.14/bin/pig /opt/mapr/bin/
ln -sf /opt/mapr/pig/pig-0.14/bin/pig /usr/bin/pig


#
# single pig jar for post-yarn releases
#
if [ -f /opt/mapr/MapRBuildVersion ]; then
    MAPR_VERSION=`cat /opt/mapr/MapRBuildVersion | awk -F "." '{print $1"."$2}'`

    #if mapr-core release >=4.0 (yarn beta) returns boolean 1, else returns boolean 0
    POST_YARN=`echo | awk -v cur=$MAPR_VERSION -v min=4.0 '{if (cur >= min) printf("1"); else printf ("0");}'`
    if [[ $POST_YARN == 0 ]]; then
echo POST_YARN=$POST_YARN, removing yarn jars    
        rm -f  /opt/mapr/pig/pig-0.14/pig-core-h1.jar
        mv /opt/mapr/pig/pig-0.14/pig-*-h1.jar /opt/mapr/pig/pig-0.14/pig-core-h1.jar
        mv /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h1.jar /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank.jar
        rm /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h2.jar
        rm /opt/mapr/pig/pig-0.14/pig-*-h2.jar
        mv /opt/mapr/pig/pig-0.14/conf/pig.properties.3.x /opt/mapr/pig/pig-0.14/conf/pig.properties
    else
# mv *-h2.jar to *.jar
echo POST_YARN=$POST_YARN, replacing MR1 jars
 rm -f  /opt/mapr/pig/pig-0.14/pig-core-h2.jar
        mv /opt/mapr/pig/pig-0.14/pig-*-h2.jar /opt/mapr/pig/pig-0.14/pig-core-h2.jar
        mv /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h2.jar /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank.jar
        rm /opt/mapr/pig/pig-0.14/contrib/piggybank/java/piggybank-h1.jar
        rm /opt/mapr/pig/pig-0.14/pig-*-h1.jar
        rm /opt/mapr/pig/pig-0.14/conf/pig.properties.3.x
    fi    
fi

#
# change ownership
#
DAEMON_CONF="/opt/mapr/conf/daemon.conf"

if [ -f "$DAEMON_CONF" ]; then
    MAPR_USER=$( awk -F = '$1 == "mapr.daemon.user" { print $2 }' $DAEMON_CONF)

    if [ ! -z "$MAPR_USER" ]; then
        chown -R $MAPR_USER /opt/mapr/pig/pig-0.14
    fi
fi

##
## end of post install section
##
preuninstall program: /bin/sh
postuninstall scriptlet (using /bin/sh):

#
# last version is removed
# if package is getting purged remove entire directory
# we may need to remove pig/pig-X/logs
#
if [ "$1" = "0" ]; then
    rm -Rf /opt/mapr/pig/pig-0.14/logs
    rm -Rf /opt/mapr/pig/pig-0.14/conf

    rm -f /opt/mapr/bin/pig
    rm -f /usr/bin/pig
fi

##
## end of post uninstall section
##
[root@node1 pig-0.14]# 

           

                           Run Pig Jobs with Oozie          

This Blog assumes below Oozie and Pig package are already installed on running MapR cluster and Steps from blog 1 is already followed.

mapr-oozie-4.1.0.201606271017-1.noarch
mapr-oozie-internal-4.1.0.201606271017-1.noarch
mapr-pig-0.14.201608040131-1.noarch

http://abizeradenwala.blogspot.com/2015/07/installing-oozie-and-running-sample-job.html

Since Oozie is current bundled with Pig v0.12 we will need below steps for oozie pig action to work.

/opt/mapr/oozie/oozie-4.1.0/share1/lib/pig/pig-withouthadoop-0.12.1-mapr-1408-h2.jar
/opt/mapr/oozie/oozie-4.1.0/share1/lib/pig-2/pig-withouthadoop-0.12.1-mapr-1408-h2.jar

1) The Oozie share/lib directory has two sets of JAR files for Pig.  We will use the Pig JAR files from the share/lib/pig-2 directory with MapR distribution versions 4.0.0 and later.

To specify the JAR files for a given Pig job, add the following section to the workflow.xml file:

<name>oozie.action.sharelib.for.pig</name>

<value>pig-2</value>

2) Stop Oozie:

maprcli node services -name oozie -action stop -nodes <nodes>

3)  Remove all files located within the /opt/mapr/oozie/oozie<version>/share2/lib/pig*/ directory EXCEPT the oozie-sharelib-pig-<version>-mapr.jar file.
Now copy new Pig jars to share lib location,

cp <PIG_HOME>/pig-core-h2.jar <OOZIE_HOME>/share2/lib/pig-2/

cp <PIG_HOME>/lib/* <OOZIE_HOME>/share2/lib/pig-2/ 

4) Remove the zookeeper jars .

rm -rf <OOZIE_HOME>/share2/lib/pig-2/zookeeper*.jar

5) Now move all the old jars in latest share lib in MaprFS to temp location.

hadoop fs -mv /oozie/share/lib/lib_20160804181903/pig-2/* /abizer

And now copy latest jars into share lib in MaprFS

hadoop fs -put /opt/mapr/oozie/oozie-4.1.0//share2/lib/pig-2/*   /oozie/share/lib/lib_20160804181903/pig-2

6) Copy work-flow.xml to maprfs which is specified in job.properties file

hadoop fs -put workflow.xml /user/mapr/examples/apps/pig/workflow.xml

Example of my workflow.xml
[mapr@node3 pig-2]$ cat /opt/mapr/oozie/oozie-4.1.0/examples/apps/pig/workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.2" name="pig-wf">
    <start to="pig-node"/>
    <action name="pig-node">
        <pig>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
                <delete path="${nameNode}/user/${wf:user()}/output-data/pig"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.compress.map.output</name>
                    <value>true</value>
                </property>
                <property>
                        <name>oozie.action.sharelib.for.pig</name>
                        <value>pig-2</value>
                 </property>
            </configuration>
            <script>id.pig</script>
            <param>INPUT=/user/${wf:user()}/input-data/text</param>
            <param>OUTPUT=/user/${wf:user()}/output-data/pig</param>
        </pig>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Pig failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

7 ) Start Oozie:

maprcli node services -name oozie -action start -nodes <nodes>

8) As user MapR i am running sample workflow.

[mapr@node3 root]$ /opt/mapr/oozie/oozie-4.1.0/bin/oozie job -oozie="http://localhost:11000/oozie" -config /opt/mapr/oozie/oozie-4.1.0/examples/apps/pig/job.properties -run
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/mapr/oozie/oozie-4.1.0/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
job: 0000000-160805145748028-oozie-mapr-W

9) On checking the status pig wf was successfully executed by Oozie.

[mapr@node3 root]$ /opt/mapr/oozie/oozie-4.1.0/bin/oozie job -info 0000000-160805145748028-oozie-mapr-W -oozie="http://localhost:11000/oozie"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/mapr/oozie/oozie-4.1.0/lib/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mapr/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
Job ID : 0000000-160805145748028-oozie-mapr-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : pig-wf
App Path      : maprfs:/user/mapr/examples/apps/pig
Status        : SUCCEEDED
Run           : 0
User          : mapr
Group         : -
Created       : 2016-08-05 18:58 GMT
Started       : 2016-08-05 18:58 GMT
Last Modified : 2016-08-05 18:59 GMT
Ended         : 2016-08-05 18:59 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000000-160805145748028-oozie-mapr-W@:start:                                  OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------
0000000-160805145748028-oozie-mapr-W@pig-node                                 OK        job_1470423487588_0001 SUCCEEDED  -        
------------------------------------------------------------------------------------------------------------------------------------
0000000-160805145748028-oozie-mapr-W@end                                      OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------

[mapr@node3 root]$

Rename SP in MapR

                                                 Rename SP in MapR

Sometimes for consistency Admin would want SP name to be same as the old SP name even after SP is recreated. Usually after SP is recreated it doesn't have the same original SP name .


Below are the steps to rename an SP. 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp list 
ListSPs resp: status 0:2 
No. of SPs (2), totalsize 584687 MB, totalfree 583859 MB 

SP 0: name SP1, Online, size 290295 MB, free 289881 MB, path /dev/sdb 
SP 1: name SP2, Online, size 294391 MB, free 293978 MB, path /dev/sde 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp offline /dev/sde 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp list 
ListSPs resp: status 0:2 
No. of SPs (2), totalsize 290295 MB, totalfree 289881 MB 

SP 0: name SP1, Online, size 290295 MB, free 289881 MB, path /dev/sdb 
SP 1: name SP2, Offline, size 307197 MB, free 0 MB, path /dev/sde 

[root@qa-node-111 ~]# /opt/mapr/server/mrconfig sp rename -n SP4 /dev/sde 
SP renamed to SP4 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp online /dev/sde 

[root@node-1 ~]# /opt/mapr/server/mrconfig sp list 
ListSPs resp: status 0:2 
No. of SPs (2), totalsize 584687 MB, totalfree 583859 MB 

SP 0: name SP1, Online, size 290295 MB, free 289881 MB, path /dev/sdb 
SP 1: name SP4, Online, size 294391 MB, free 293977 MB, path /dev/sde 

Now SP2 is rename to SP4 as needed.