Hbase table copy across secure MapR clusters

                                   Hbase table copy across secure MapR clusters

   

This Blog assumes you have 2 clusters up and running securely

Note :-

Source Cluster A - Node 10.10.70.111

Destination Cluster B - Node 10.10.70.109  

Also, in Secure cluster we need to specify hbase to use maprsecurity when submitting the map reduce job to do so edit  /opt/mapr/conf/env.sh file as below ( All nodes).

#export MAPR_HBASE_CLIENT_OPTS="${SIMPLE_LOGIN_OPTS} -Dzookeeper.sasl.client=false"
export MAPR_HBASE_CLIENT_OPTS="${MAPR_LOGIN_OPTS}"

Stop all services on destination clusters ( Cluster B ) and perform the following steps:-

1. Merge truststore on destination . 

scp /opt/mapr/conf/ssl_truststore 10.10.70.109:/tmp/     ( Copy truststore from source (A) to destination(B) cluster )

chmod 644 /opt/mapr/conf/ssl_truststore                         ( make truststore writable )

( Merge source SSL_Truststore with destination )

/opt/mapr/server/manageSSLKeys.sh merge /tmp/ssl_truststore /opt/mapr/conf/ssl_truststore

chmod 400 /opt/mapr/conf/ssl_truststore                        ( Make truststore readonly )

2. Remove existing cldb.key , maprserverticket and 'mapruserticket' from cluster B. 

rm -f /opt/mapr/conf/cldb.key ; rm -f /opt/mapr/conf/maprserverticket; 

rm -f /opt/mapr/conf/mapruserticket

3. copy cldb.key from cluster A to B (to cldb & ZK nodes)

scp /opt/mapr/conf/cldb.key  10.10.70.109:/opt/mapr/conf/ 

4. Make a copy of 'maprserverticket' on cluster A

cp /opt/mapr/conf/maprserverticket /opt/mapr/conf/maprserverticket_copy

5. Edit copied 'maprserverticket_copy' and modify the cluster name (first portion) to name of cluster B. (retain the remaining part as is). 

i.e

$ cat /opt/mapr/conf/maprserverticket 

Source tXDbAlh2d6+TJZPjGpXcWo/LRHWqPV6iMo58iQnSUSBpvx+daM7kM64ww/Fpow934VVnH7acdMBfV2fxipno49LsSryEcC1aEnMVHFw6Sbwifrr8PURddkhrd6kEO11+JSFgFdF4qYXfmuQGZAIHX+OgORkztRhdF0AKWEPif38+OiuFXJrqxzchms/FoqUYQ9o50bTeuZ82zieG4Z6/sR/CFbtrsko8EA9pH9xHu9Od5M6PauSBWOj4J+nwpJVTOKT4PlM3kuk/0Z7IVg==

$ cat /opt/mapr/conf/maprserverticket_copy 

Dest tXDbAlh2d6+TJZPjGpXcWo/LRHWqPV6iMo58iQnSUSBpvx+daM7kM64ww/Fpow934VVnH7acdMBfV2fxipno49LsSryEcC1aEnMVHFw6Sbwifrr8PURddkhrd6kEO11+JSFgFdF4qYXfmuQGZAIHX+OgORkztRhdF0AKWEPif38+OiuFXJrqxzchms/FoqUYQ9o50bTeuZ82zieG4Z6/sR/CFbtrsko8EA9pH9xHu9Od5M6PauSBWOj4J+nwpJVTOKT4PlM3kuk/0Z7IVg==

-bash-4.1$

6. Copy the modified maprserverticket to cluster B (all nodes)

scp /opt/mapr/conf/maprserverticket_copy 10.10.70.109:/opt/mapr/conf/maprserverticket

7 Make sure the permission of cldb.key and maprserverticket are correct (owned by mapr:mapr).

[mapr@410-Dest ~]$ ls -l /opt/mapr/conf/cldb.key 

-rw------- 1 mapr mapr 89 Mar 18 15:51 /opt/mapr/conf/cldb.key

[mapr@410-Dest ~]$ ls -l /opt/mapr/conf/maprserverticket 

-rw------- 1 mapr mapr 282 Mar 18 16:11 /opt/mapr/conf/maprserverticket

8 Start zookeeper & warden on respective nodes. 

9 Make sure the services are up & running on cluster B and attempt CopyTable from source.

Run copy table command on source cluster :

 hbase org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=10.10.70.109:5181:/hbase weblog

2016-03-18 16:28:39,679 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

2016-03-18 16:28:39,874 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-mapr-1503--1, built on 03/26/2015 18:33 GMT

2016-03-18 16:28:39,874 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_45

2016-03-18 16:28:39,887 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=mapr

2016-03-18 16:28:39,887 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/mapr

2016-03-18 16:28:39,887 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/mapr/zookeeper/zookeeper-3.4.5/logs

2016-03-18 16:28:39,888 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.10.70.109:5181 sessionTimeout=30000 watcher=com.mapr.util.zookeeper.ZKDataRetrieval@2e14ec19

2016-03-18 16:28:39,914 INFO  [main-SendThread(410-Dest:5181)] zookeeper.Login: successfully logged in.

2016-03-18 16:28:39,915 INFO  [main-SendThread(410-Dest:5181)] client.ZooKeeperSaslClient: Client will use MAPR-SECURITY as SASL mechanism.

2016-03-18 16:28:39,918 INFO  [main-SendThread(410-Dest:5181)] zookeeper.ClientCnxn: Opening socket connection to server 410-Dest/10.10.70.111:5181. Will attempt to SASL-authenticate using Login Context section 'Client'

2016-03-18 16:28:39,944 INFO  [main-SendThread(410-Dest:5181)] zookeeper.ClientCnxn: Session establishment complete on server 410-Dest/10.10.70.109:5181, sessionid = 0x1538c23bd930050, negotiated timeout = 30000

2016-03-18 16:28:39,947 INFO  [main] zookeeper.ZKDataRetrieval: Connected to ZK: 10.10.70.111:5181

2016-03-18 16:28:39,947 INFO  [main] zookeeper.ZKDataRetrieval: Getting serviceData for master node of resourcemanager

2016-03-18 16:28:39,965 INFO  [main-EventThread] zookeeper.ZKDataRetrieval: Process path: null. Event state: SaslAuthenticated. Event type: None

2016-03-18 16:28:39,975 INFO  [main] client.MapRZKBasedRMFailoverProxyProvider: Updated RM address to 410-Dest/10.10.70.109:8032

2016-03-18 16:28:40,279 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

2016-03-18 16:28:40,373 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x74727a94 connecting to ZooKeeper ensemble=10.10.70.111:5181

2016-03-18 16:28:40,374 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.10.70.111:5181 sessionTimeout=90000 watcher=hconnection-0x74727a940x0, quorum=10.10.70.111:5181, baseZNode=/hbase

2016-03-18 16:28:40,375 INFO  [main-SendThread(410-Source:5181)] client.ZooKeeperSaslClient: Client will use MAPR-SECURITY as SASL mechanism.

2016-03-18 16:28:40,376 INFO  [main-SendThread(410-Source:5181)] zookeeper.ClientCnxn: Opening socket connection to server 410-Source/10.10.70.111:5181. Will attempt to SASL-authenticate using Login Context section 'Client'

2016-03-18 16:28:40,383 INFO  [main-SendThread(410-Source:5181)] zookeeper.ClientCnxn: Socket connection established to 410-Source/10.10.70.111:5181, initiating session

2016-03-18 16:28:40,409 INFO  [main-SendThread(410-Source:5181)] zookeeper.ClientCnxn: Session establishment complete on server 410-Source/10.10.70.111:5181, sessionid = 0x1538bc340af0076, negotiated timeout = 40000

2016-03-18 16:28:40,435 INFO  [main] mapreduce.TableOutputFormat: Created table instance for weblog

2016-03-18 16:28:41,032 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x20828fe4 connecting to ZooKeeper ensemble=10.10.70.109:5181

2016-03-18 16:28:41,032 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=10.10.70.109:5181 sessionTimeout=90000 watcher=hconnection-0x20828fe40x0, quorum=10.10.70.109:5181, baseZNode=/hbase

2016-03-18 16:28:41,033 INFO  [main-SendThread(410-Source:5181)] client.ZooKeeperSaslClient: Client will use MAPR-SECURITY as SASL mechanism.

2016-03-18 16:28:41,033 INFO  [main-SendThread(410-Source:5181)] zookeeper.ClientCnxn: Opening socket connection to server 410-Dest/10.10.70.111:5181. Will attempt to SASL-authenticate using Login Context section 'Client'

2016-03-18 16:28:41,034 INFO  [main-SendThread(410-Source:5181)] zookeeper.ClientCnxn: Socket connection established to 410-Dest/10.10.70.111:5181, initiating session

2016-03-18 16:28:41,067 INFO  [main-SendThread(410-Source:5181)] zookeeper.ClientCnxn: Session establishment complete on server 410-Source/10.10.70.111:5181, sessionid = 0x1538c23bd930051, negotiated timeout = 40000

2016-03-18 16:28:41,115 INFO  [main] util.RegionSizeCalculator: Calculating region sizes for table "weblog".

2016-03-18 16:28:41,500 INFO  [main] mapreduce.JobSubmitter: number of splits:1

2016-03-18 16:28:41,511 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum

2016-03-18 16:28:41,765 INFO  [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1458345179933_0003

2016-03-18 16:28:41,964 INFO  [main] security.ExternalTokenManagerFactory: Initialized external token manager class - com.mapr.hadoop.yarn.security.MapRTicketManager

2016-03-18 16:28:42,001 INFO  [main] impl.YarnClientImpl: Submitted application application_1458345179933_0003

2016-03-18 16:28:42,063 INFO  [main] mapreduce.Job: The url to track the job: https://410-Source:8090/proxy/application_1458345179933_0003/

2016-03-18 16:28:42,064 INFO  [main] mapreduce.Job: Running job: job_1458345179933_0003

2016-03-18 16:28:53,332 INFO  [main] mapreduce.Job: Job job_1458345179933_0003 running in uber mode : false

2016-03-18 16:28:53,334 INFO  [main] mapreduce.Job:  map 0% reduce 0%

2016-03-18 16:28:59,602 INFO  [main] mapreduce.Job:  map 100% reduce 0%

2016-03-18 16:28:59,622 INFO  [main] mapreduce.Job: Job job_1458345179933_0003 completed successfully

2016-03-18 16:28:59,786 INFO  [main] mapreduce.Job: Counters: 41

            File System Counters

                        FILE: Number of bytes read=0

                        FILE: Number of bytes written=111880

                        FILE: Number of read operations=0

                        FILE: Number of large read operations=0

                        FILE: Number of write operations=0

                        MAPRFS: Number of bytes read=66

                        MAPRFS: Number of bytes written=0

                        MAPRFS: Number of read operations=11

                        MAPRFS: Number of large read operations=0

                        MAPRFS: Number of write operations=0

            Job Counters 

                        Launched map tasks=1

                        Data-local map tasks=1

                        Total time spent by all maps in occupied slots (ms)=4698

                        Total time spent by all reduces in occupied slots (ms)=0

                        Total time spent by all map tasks (ms)=4698

                        Total vcore-seconds taken by all map tasks=4698

                        Total megabyte-seconds taken by all map tasks=4810752

                        DISK_MILLIS_MAPS=2349

            Map-Reduce Framework

                        Map input records=2

                        Map output records=2

                        Input split bytes=66

                        Spilled Records=0

                        Failed Shuffles=0

                        Merged Map outputs=0

                        GC time elapsed (ms)=51

                        CPU time spent (ms)=1400

                        Physical memory (bytes) snapshot=223748096

                        Virtual memory (bytes) snapshot=1834762240

                        Total committed heap usage (bytes)=159383552

            HBase Counters

                        BYTES_IN_REMOTE_RESULTS=0

                        BYTES_IN_RESULTS=102

                        MILLIS_BETWEEN_NEXTS=562

                        NOT_SERVING_REGION_EXCEPTION=0

                        NUM_SCANNER_RESTARTS=0

                        REGIONS_SCANNED=1

                        REMOTE_RPC_CALLS=0

                        REMOTE_RPC_RETRIES=0

                        RPC_CALLS=3

                        RPC_RETRIES=0

            File Input Format Counters 

                        Bytes Read=0

            File Output Format Counters 

                                   Bytes Written=0

Yum

                                                                  YUM 

Yum is an interactive, rpm-based package manager. It can automatically perform system updates, including dependency analysis and processing based on "repository" metadata. It can perform installation of new packages, removal of old packages and perform queries on the installed and/or available packages among many other commands available.

The configuration file for Yum and related utilities is located at /etc/yum.conf . All the repositories new or existing are with .repo file extension under /etc/yum.repo.d/ directory.

For example I created below repo  :

[root@node4 ~]# cat /etc/yum.repos.d/maprtech.repo

[maprtech]

name=MapR Technologies

baseurl=http://package.mapr.com/releases/v4.0.1/redhat/

enabled=1

gpgcheck=0

protect=1

Here  :

Name - Human readable name of the repository

Baseurl - URL to the repodata directory ( I.e where all the rpm's are hosted )

Enabled - Enable repository when performing updates and installs (enabled=1)

Gpgcheck - Enable/disable GPG signature checking (gpgcheck=0  disables GPG-checking)

Protect - To protect a repository ( mainly protect existing package being updated from existing repo by package in Non-standard repo )

Some Common examples and tricks to use Yum :

1) List all spark packages available from repo .

[root@node4 ~]# yum list | grep mapr-spark

mapr-spark.noarch                           1.5.2.201512161339-1         maprecosystem

mapr-spark-historyserver.noarch             1.5.2.201512161339-1         maprecosystem

mapr-spark-master.noarch                    1.5.2.201512161339-1         maprecosystem

2) Erase installed/erase package with dependencies .

yum install mapr-spark  ( This will only install latest Spark version if multiple avail in Repo )

yum erase mapr-spark 

3) If you want to install specific version of spark .

a)  List all Spark package version.

[root@node4 ~]# yum --showduplicates list mapr-spark

Available Packages

mapr-spark.noarch                                                                1.0.2.201408081440-1                                                                maprecosystem 

mapr-spark.noarch                                                                1.1.0.201411081007-1                                                                maprecosystem 

mapr-spark.noarch                                                                1.2.1.201506091827-1                                                                maprecosystem 

mapr-spark.noarch                                                                1.3.1.201506011823-1                                                                maprecosystem 

mapr-spark.noarch                                                                1.4.1.201509021818-1                                                                maprecosystem 

mapr-spark.noarch                                                                1.5.2.201512161339-1                                                                maprecosystem 

b) Now if you like to install specific version of package you can append the version info to the name of the package as  yum install <package name>-<version info>

yum install mapr-spark-1.2.1.201506091827-1   ( This will install spark-1.2 instead of latest)

4) Now incase you have some repo with outdated links causing issues , we can ignore all other repo except MAPR_CORE via below command.

yum --disablerepo=* --enablerepo=MapR_CORE install mapr-fileserver 

Identify SSD drives V/S Hard disks

                                      Identify SSD drives V/S Hard disks

Quick way to identify if disks are SSD's or Hard Drives by viewing the content of /sys/block/sd*/queue/rotational file, If this file has value 1 it indicates its Hard Drive while if it has 0 for SSD's .

Like in below example sda is hard drive while sdb-sdg are SSD drives .

[root@tss2-18 ~]# ls -l /sys/block/sd*/queue/rotational
-rw-r--r-- 1 root root 4096 Jan 25 23:08 /sys/block/sda/queue/rotational
-rw-r--r-- 1 root root 4096 Dec 25 22:02 /sys/block/sdb/queue/rotational
-rw-r--r-- 1 root root 4096 Dec 25 22:02 /sys/block/sdc/queue/rotational
-rw-r--r-- 1 root root 4096 Dec 25 22:02 /sys/block/sdd/queue/rotational
-rw-r--r-- 1 root root 4096 Dec 25 22:02 /sys/block/sde/queue/rotational
-rw-r--r-- 1 root root 4096 Dec 25 22:02 /sys/block/sdf/queue/rotational
-rw-r--r-- 1 root root 4096 Dec 25 22:02 /sys/block/sdg/queue/rotational
[root@tss2-18 ~]# cat /sys/block/sd*/queue/rotational
1
0
0
0
0
0
0

Another way to find this out would be via "lsblk" command as below .

[root@tss2-18 ~]# lsblk -d -o name,rota
NAME ROTA
sdb     0
sdc     0
sdd     0
sde     0
sdf     0
sdg     0
sda     1

Note :  ROTA means rotational device  ( 1 if true, 0 if false)

For above command to work we have to make sure "util-linux" package is installed which provides "lsblk" utility .

[root@tss2-18 ~]# which lsblk
/bin/lsblk
[root@tss2-18 ~]# rpm -qf /bin/lsblk
util-linux-ng-2.17.2-12.18.el6.x86_64

Performance Issue / Hang issue

                                                Performance Issue / Hang issue

-First we can use vmstat to check system resource utilization and see which resource is scarce when hang is observed as below. If its due to Memory constrains or system swapping its easy to address by adding more memory to the server or putting cap on processes to utilize only dedicated amount of memory. Below blog gives detail on identifying system resource constrains step by step.

http://abizeradenwala.blogspot.com/2015/07/resource-utilization-to-quickly.html

- If we identify its certain process causing it then we can start with htop to identify if processes hogging most amount of resources have multiple threads running in them internally . Htop is usually not installed on the system by default and might need below package installed.

htop-1.0.1-2.el6.x86_64

Using htop, press t to get a nested tree of threads and specifically look at PID which seems to be hogging most amount of resources.

Alternatively to get list of all the threads and processes currently running on the system we would need to run below command to get all threads .  Running below ps as a cron job and poll the system every 5 mins and recreate the system hang can identify if there are huge number of threads which have been spunned up via java internally.

ps -eLf | grep 21917


UID        PID  PPID   LWP  C NLWP   STIME TTY           TIME     CMD

mapr     21887     1    21917  0   53           Sep11                   00:01:05   /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java

  -L  Show threads, possibly with LWP( light wt process) and NLWP ( Not light wt process ) columns

- If we do see some process continuously hogging CPU it would point us to some part of Code is not using CPU effectively or some threads are stuck in CPU and not making progress . We should run 

"kill -3 " every 1 sec for it to thread dump to standard out of the process and carefully review the thread dump with Developers to identify any issue in the code. This will usually show which thread is slow or not making any or significant progress during the time timeframe when issue was observed , this would help developers to put the fix in the code to avoid potential hang.

Iostat

                                                                     Iostat


Alot of time we need to find out disk IO utilization, measure if all disks are performing well and monitor system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates.  I below example i ran dd on disk "sdb" and ran iostat in back ground, its seen disk is busy i.e %util is almost 100%.

dd bs=1M count=4096 if=/dev/zero of=/dev/sdb oflag=direct    ( Directly writing to disk )

Iostat was run with below options where m = display numbers in MB  -t = print time stamp
-x = Display extended statistics

 iostat -m -t -x 1

09/11/2015 02:28:39 PM
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.00    0.00      2.27      41.48      0.00   56.25
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00          0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdb               0.00     0.00          0.00   44.00     0.00    22.00  1024.00     1.57   34.82  22.68  99.80
sdc               0.00     0.00          0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd               0.00     0.00          0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sde               0.00     0.00          0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00



Line 1 : On First line it prints time
Line 2 and 3 : Print CPU stats/utilization , here we see CPU is still idle

Line 5 --- n : Prints various stats for each disk ( Stat printed mean as below ) , I will shortly explain which once are what and what to infer out of this numbers.

• rrqm/s : The number of read requests merged per second that were queued to the hard disk
• wrqm/s : The number of write requests merged per second that were queued to the hard disk
• r/s : The number of read requests per second
• w/s : The number of write requests per second
• rsec/s : The number of sectors read from the hard disk per second
• wsec/s : The number of sectors written to the hard disk per second
• avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
• avgqu-sz : The average queue length of the requests that were issued to the device If one complains about I/O performance issues when avgqu-sz is lower, then it is application specific stuff, that can be resolved with more aggressive read-ahead, less fsyncs, etc. One interesting part – avqu-sz, await, svctm and %util are iterdependent ( await = avgqu-sz * svctm / (%util/100)
• await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. If there is not alot of I/O being created but there are requests just pending it could be due to disk being slow due to H/W issue.
• svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
• %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. Also this value is excluding any kind of cache here – if request can be served from cache, the chance is quite negligible it will show up in %util, unlike in other values. 

If below values from the iostat output is high it would mean the specific disk is under pressure: Clearly in above example sdb is being utilized a lot.

1. The average service time (svctm)
2. Percentage of CPU time during which I/O requests were issued (%util)
3. If a hard disk reports consistently high reads/writes (r/s and w/s)
4. await is continuously high  ( Very important ) . 
5. avgqu-sz is continuously high ( very important )

Note :- " -n " option Displays the network filesystem (NFS) report

commonly accepted averages

Rotational Speed (rpm)	IOPS
5400	50-80
7200	75-100
10k	125-150
15k	175-210

Top Command Explanation

                                                         Top Command Explanation 

As a Linux system admin top command is a frequently used command to view resource utilization (Memory and CPU) by processes on server. This command helps us to find which process is utilizing what resources of system and nail down the process which is hogging all memory or churning CPU.

Although there are much better and user friendly tools then top like htop but i would consider this to be used as a first level command and later use more specific commands to further drill down to RC the issue . Below post i am writing on how to use and read results of top command.

Reading Linux Top Command Output:

When we execute top command on linux, it shows a lot of results, here i am trying to show you to how to read it row by row.

Result Row #1:

Row 1 results shows about server up time from last reboot, currently logged in users and cpu load on server. The same output you can find using linux uptime command.

top - 21:56:08 up 62 days,  6:38,  3 users,  load average: 0.08, 0.04, 0.00

Result Row #2:

Row 2 shows the number of process running on server and there state.

Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie

Zombie process is a process that has completed execution but still has an entry in the process table. This entry is still needed to allow the parent process to read its child’s exit status. Zombies are basically the leftover bits of dead processes that haven’t been cleaned up properly. A program that creates zombie processes isn’t programmed properly – programs aren’t supposed to let zombie processes stick around.

Result Row #3:

Row three shows the cpu utilization status on server, you can find here how much cpu is free and how much is utilizing by system.

Cpu(s):  0.3%us,  0.5%sy,  0.0%ni, 98.8%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st

0.3% us : %CPU used by User processes

0.5%sy : %CPU used by System Processes

0.0%ni : %CPU used by setting Nice value
98.8%id : %CPU in Idle state
0.3%wa : %CPU Waiting on I/O
0.0%hi / 0.0%si  :  %CPU used by Hardware/Software Interrupts 
0.0%st : Steal time is the time that a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.

Result Row #4:

Row 4 shows the memory utilization on server, you can find here how much memory is used, the same results you can find using free command.

            Total Memory    Used Memory      Free Memory    Buffered Memory

Mem:   8193720k total,  6317764k used,   1875956k free,   493816k buffers

Result Row #5:

Row 5 shows the swap memory utilization on server, you can find here how much swap is being used, the same results you can find using free command ( last line ). Usually if you system is swapping / using swap memory it indicated system is under memory pressure and will most likely cause system to run very slow

            Total Swap Mem   Used Swap      Free Swap        Cached Swap

Swap:  9215992k total,      3920k used,  9212072k free,  1742556k cached

Result Row #6 ( Running Processes ):

In this steps you will see all running process on servers and details about each process as below.

PID USER      PR  NI  VIRT   RES  SHR S %CPU %MEM     TIME+   COMMAND                                                                                                             4706 mapr        10 -10  2706m   2.4g  17m  S    2.0       30.7      641:03.31    mfs    

PID - Process ID of process           USER - User who's running the process

PR - Priority of process                 NI - Nice value of process

VIRT - Virtual Memory used by process    RES - Physical Memory used by process ( Actual Memory )

SHR - Shared Memory used by process     S -  Current status of the process

%CPU / %MEM - % CPU/MEM used by the process

TIME+   -  Total time process is been running for.         COMMAND - Name of process

By default TOP sorts the output by %CPU usage (k), If you would want to sort output on basis of any fields, you can use SHIFT+F and select appropriate Alphabet to sort field via field as below and press Enter.

 a: PID        = Process Id

  b: PPID       = Parent Process Pid

  c: RUSER      = Real user name

  d: UID        = User Id

  e: USER       = User Name

  f: GROUP      = Group Name

  g: TTY        = Controlling Tty

  h: PR         = Priority

  i: NI         = Nice value

  j: P          = Last used cpu (SMP)

* K: %CPU       = CPU usage

  l: TIME       = CPU Time

  m: TIME+      = CPU Time, hundredths

  n: %MEM       = Memory usage (RES)

  o: VIRT       = Virtual Image (kb)

  p: SWAP       = Swapped size (kb)

  q: RES        = Resident size (kb)

  r: CODE       = Code size (kb)

  s: DATA       = Data+Stack size (kb)

  t: SHR        = Shared Mem size (kb)

  u: nFLT       = Page Fault count

  v: nDRT       = Dirty Pages count

  w: S          = Process Status

  x: COMMAND    = Command name/line

  y: WCHAN      = Sleeping in Function

 z: Flags      = Task Flags <sched.h>

Note :-  When top is running and you press 1 it will show you per CPU (different for each CPU) resource utilization .