Top Command Explanation

                                                         Top Command Explanation 

As a Linux system admin top command is a frequently used command to view resource utilization (Memory and CPU) by processes on server. This command helps us to find which process is utilizing what resources of system and nail down the process which is hogging all memory or churning CPU.

Although there are much better and user friendly tools then top like htop but i would consider this to be used as a first level command and later use more specific commands to further drill down to RC the issue . Below post i am writing on how to use and read results of top command.

Reading Linux Top Command Output:

When we execute top command on linux, it shows a lot of results, here i am trying to show you to how to read it row by row.

Result Row #1:

Row 1 results shows about server up time from last reboot, currently logged in users and cpu load on server. The same output you can find using linux uptime command.

top - 21:56:08 up 62 days,  6:38,  3 users,  load average: 0.08, 0.04, 0.00

Result Row #2:

Row 2 shows the number of process running on server and there state.

Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie

Zombie process is a process that has completed execution but still has an entry in the process table. This entry is still needed to allow the parent process to read its child’s exit status. Zombies are basically the leftover bits of dead processes that haven’t been cleaned up properly. A program that creates zombie processes isn’t programmed properly – programs aren’t supposed to let zombie processes stick around.

Result Row #3:

Row three shows the cpu utilization status on server, you can find here how much cpu is free and how much is utilizing by system.

Cpu(s):  0.3%us,  0.5%sy,  0.0%ni, 98.8%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st

0.3% us : %CPU used by User processes

0.5%sy : %CPU used by System Processes

0.0%ni : %CPU used by setting Nice value
98.8%id : %CPU in Idle state
0.3%wa : %CPU Waiting on I/O
0.0%hi / 0.0%si  :  %CPU used by Hardware/Software Interrupts 
0.0%st : Steal time is the time that a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor.

Result Row #4:

Row 4 shows the memory utilization on server, you can find here how much memory is used, the same results you can find using free command.

            Total Memory    Used Memory      Free Memory    Buffered Memory

Mem:   8193720k total,  6317764k used,   1875956k free,   493816k buffers

Result Row #5:

Row 5 shows the swap memory utilization on server, you can find here how much swap is being used, the same results you can find using free command ( last line ). Usually if you system is swapping / using swap memory it indicated system is under memory pressure and will most likely cause system to run very slow

            Total Swap Mem   Used Swap      Free Swap        Cached Swap

Swap:  9215992k total,      3920k used,  9212072k free,  1742556k cached

Result Row #6 ( Running Processes ):

In this steps you will see all running process on servers and details about each process as below.

PID USER      PR  NI  VIRT   RES  SHR S %CPU %MEM     TIME+   COMMAND                                                                                                             4706 mapr        10 -10  2706m   2.4g  17m  S    2.0       30.7      641:03.31    mfs    

PID - Process ID of process           USER - User who's running the process

PR - Priority of process                 NI - Nice value of process

VIRT - Virtual Memory used by process    RES - Physical Memory used by process ( Actual Memory )

SHR - Shared Memory used by process     S -  Current status of the process

%CPU / %MEM - % CPU/MEM used by the process

TIME+   -  Total time process is been running for.         COMMAND - Name of process

By default TOP sorts the output by %CPU usage (k), If you would want to sort output on basis of any fields, you can use SHIFT+F and select appropriate Alphabet to sort field via field as below and press Enter.

 a: PID        = Process Id

  b: PPID       = Parent Process Pid

  c: RUSER      = Real user name

  d: UID        = User Id

  e: USER       = User Name

  f: GROUP      = Group Name

  g: TTY        = Controlling Tty

  h: PR         = Priority

  i: NI         = Nice value

  j: P          = Last used cpu (SMP)

* K: %CPU       = CPU usage

  l: TIME       = CPU Time

  m: TIME+      = CPU Time, hundredths

  n: %MEM       = Memory usage (RES)

  o: VIRT       = Virtual Image (kb)

  p: SWAP       = Swapped size (kb)

  q: RES        = Resident size (kb)

  r: CODE       = Code size (kb)

  s: DATA       = Data+Stack size (kb)

  t: SHR        = Shared Mem size (kb)

  u: nFLT       = Page Fault count

  v: nDRT       = Dirty Pages count

  w: S          = Process Status

  x: COMMAND    = Command name/line

  y: WCHAN      = Sleeping in Function

 z: Flags      = Task Flags <sched.h>

Note :-  When top is running and you press 1 it will show you per CPU (different for each CPU) resource utilization .

How to Install, configure and use Clush

                                        How to Install, configure and use Clush

Clush is an open source tool that allows you to execute commands in parallel across the nodes in the cluster. This blog describes how to install clush, configure and use clush to run commands on multiple nodes in parallel.

This blog assumes you have ssh trust created from node 1 (10.10.70.106) to rest of nodes (node2-node4) .

The clush utility need to be only installed on one node, usually the primary node (10.10.70.106) in the cluster where we will run commands in parallel and gather stats .

Installation :

Installation of clush utility is simple and straightforward, i.e enable EPEL repo ( as per your OS version) and yum install the package as listed in 3 steps below.

1)  wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

2)  rpm -Uvh epel-release-6*.rpm

3)  yum install clustershell -y 

Configuration :

I have a 4 node mapr cluster as listed below. 

[root@node1 ~]# maprcli node list -columns configuredservice

hostname  configuredservice                                ip          
node1     tasktracker,webserver,cldb,fileserver,hoststats  10.10.70.106
node2     tasktracker,cldb,fileserver,hoststats            10.10.70.107
node3     fileserver,tasktracker,hoststats,jobtracker      10.10.70.108
node4     fileserver,tasktracker,hoststats,jobtracker      10.10.70.109
[root@node1 ~]#

Now edit "/etc/clustershell/groups" file to group the nodes which has same services installed.

[root@node1 ~]# cat /etc/clustershell/groups
all: node[1-4]
zk: 10.10.70.106 10.10.70.107 10.10.70.108
jt: 10.10.70.108,10.10.70.109
web: 10.10.70.106
cldb: 10.10.70.106,10.10.70.107
[root@node1 ~]#

Execution/Utilization :

1) To run the command in parallel on all 4 nodes

[root@node1 ~]# clush -a date

node1: Mon Jul  6 14:26:37 PDT 2015
node4: Mon Jul  6 14:26:37 PDT 2015
node3: Mon Jul  6 14:26:37 PDT 2015
node2: Mon Jul  6 14:26:37 PDT 2015

[root@node1 ~]#

2) To get cleaner way output where same output for nodes is grouped.

[root@node1 ~]# clush -ab date
---------------
node[1-4] (4)
---------------
Mon Jul  6 14:28:41 PDT 2015
[root@node1 ~]#

3) To get into interactive cluster shell and get ouput for all nodes you can use "clush -a"


[root@node1 ~]# clush -a
Enter 'quit' to leave this interactive mode
Working with nodes: node[1-4]
clush> date
node1: Mon Jul  6 14:29:42 PDT 2015
node3: Mon Jul  6 14:29:42 PDT 2015
node2: Mon Jul  6 14:29:42 PDT 2015
node4: Mon Jul  6 14:29:42 PDT 2015
clush> uname -a
node2: Linux node2.mycluster.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
node4: Linux node4.mycluster.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
node1: Linux node1.mycluster.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
node3: Linux node3.mycluster.com 2.6.32-431.el6.x86_64 #1 SMP Fri Nov 22 03:15:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
clush>

4) To copy file /root/abi to all cluster nodes .

[root@node1 ~]# clush -a -c /root/abi 

[root@node1 ~]#

Verify :

[root@node1 ~]# clush -a ls /root/abi
node1: /root/abi
node4: /root/abi
node3: /root/abi
node2: /root/abi
[root@node1 ~]#

5) To remove file /root/abi to all cluster nodes


[root@node1 ~]# clush -a rm -rf /root/abi
[root@node1 ~]#

Verify :

[root@node1 ~]#clush -a ls /root/abi
node1: ls: cannot access /root/abi: No such file or directory
clush: node1: exited with exit code 2
node3: ls: cannot access /root/abi: No such file or directory
clush: node3: exited with exit code 2
node2: ls: cannot access /root/abi: No such file or directory
clush: node2: exited with exit code 2
node4: ls: cannot access /root/abi: No such file or directory
clush: node4: exited with exit code 2
[root@node1 ~]#

6)  To copy the file to different destination on nodes via clush and then verify.

[root@node1 ~]# clush -a -c /root/abi --dest /tmp
[root@node1 ~]# clush -a ls /tmp/abi
node2: /tmp/abi
node1: /tmp/abi
node4: /tmp/abi
node3: /tmp/abi
[root@node1 ~]#

7) To run specific command on group of nodes described in "/etc/clustershell/groups" file.


[root@node1 ~]# clush -g zk date
10.10.70.106: Mon Jul  6 14:40:24 PDT 2015
10.10.70.108: Mon Jul  6 14:40:24 PDT 2015
10.10.70.107: Mon Jul  6 14:40:24 PDT 2015
[root@node1 ~]# clush -g jt date
10.10.70.108: Mon Jul  6 14:40:33 PDT 2015
10.10.70.109: Mon Jul  6 14:40:33 PDT 2015
[root@node1 ~]#

Creating SSH trust quickly

                                                        Creating SSH trust quickly


For various reasons we would need to create ssh trust between 2 nodes for specific user Id's in this post I will describe a quick way to create trust using "ssh-copy-id". We are using "ssh-copy-id" since this will save significant time and effort to copy keys around to create the trust.

1) Run below command to check lists of public key parameters of all identities currently represented by the agent


ssh-add -L

If you get below output it means agent doesn't have public key for the user.


[root@node1 ~]# ssh-add -L
Could not open a connection to your authentication agent.
[root@node1 ~]#

2) Generate rsa keys for the user via below step and ensure rsa key is generated under ".ssh" folder .

[root@node1 .ssh]# ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa): 

Enter passphrase (empty for no passphrase): 

Enter same passphrase again: 

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

fc:e6:d0:05:bd:b2:8e:c3:fb:05:31:e9:ec:ef:fa:81 root@node1.mycluster.com

The key's randomart image is:

+--[ RSA 2048]----+

|                 |

|           o     |

|          = .    |

|       . o + .   |

|        S = o    |

|         + *     |

|       .. E o    |

|        o* o .   |

|        o+=++    |

+-----------------+

[root@node1 .ssh]# ls

id_rsa  id_rsa.pub  known_hosts

[root@node1 .ssh]#

3) Now add RSA identities to authentication agent and then verify public key parameters for user is currently represented by the agent

[root@node1 .ssh]# ssh-add

Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)

[root@node1 .ssh]# ssh-add -L

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwJMQfl9uq0saggFvyEkRE9+FfSAcTjhrKm/OCrG3zudOF5S63W0SSTmn6LdjKspIN+4pewVswVCOfAvbAeHojZu//5020kQt+OY76TvtUFU/YXFaKJElqKs9x9UvPCnCkIDDCJaaYC9nwAbuK6gEf78rqVIbN2uCrKyeA8WNg6EFnaFX/uCnxuikRYZCo88008KnAjvI1guF8AjDRfodJBnrYVdKRaxgwY2VVZ8MkEquTAVRnR2emnGcnZuwy4+4mByMzPKu5c+0zqB/vCFAvVL8uqB8RpEzuz91knenNtYB5sYyFdr0QEZXNTrlAOPJ1gYRHoWEn9Gdqd4jqtyyvw== /root/.ssh/id_rsa

NOTE: - If for some specific reason your ssh shell cannot make connection with authentication agent use below command which will start the agent and pid gets assigned to it.
eval "$(ssh-agent)"


[root@node1 .ssh]# eval "$(ssh-agent)"
Agent pid 2334
[root@node1 .ssh]#

4) Use ssh-copy-id to install public key form local node in a remote machine’s authorized_keys. It will ask the password once while the trust is being created.

[root@node1 .ssh]# ssh-copy-id -i root@10.10.70.108

root@10.10.70.108's password: 

Now try logging into the machine, with "ssh 'root@10.10.70.108'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

[root@node1 .ssh]#

5) Verify trust does work.  Below we can see we can directly ssh to 10.10.70.108 node without entering password.

[root@node1 .ssh]# ssh 10.10.70.108

Last login: Mon Jul  6 12:21:25 2015 from node1

[root@node3 ~]#

System Resource utilization

                         System Resource utilization

  To quickly overview of the system you would need a tool to give you some system stats which you can read and interpret which resource is causing the bottleneck, Vmstat is good tool which reports information about processes, memory, swap I/O, block IO, system, and cpu activity in just one line.   There are other tool which report system utilization stats as well but reason i prefer vmstat is because the output of vmstat command is easy to read and can be used effectively for preliminary check to help identify any system bottlenecks but later to gain more insight into suspected issues, a different kind of tool is required i.e a tool capable of more in-depth data collection for analysis.

  In this post we will talk about only Vmstat and what each column represents to effectively determine if issue is due to High CPU utilization/ Disk IO / system Swapping etc. 


Here is an output of vmstat command from my test node:

$ vmstat 1 3            ( Here I get stats every 1 second for 3 times only)



[root@ip-10-128-160-140 ~]# vmstat 1 3

procs -----------memory----------         ---swap-- -----io---- --system-- -----cpu-----

 r  b   swpd   free   buff             cache   si   so     bi    bo     in   cs    us sy id wa st

 0  0      0 23940396 178804 5127788    0    0     0     5    3      6     0 0 100  0  0

 0  0      0 23940396 178804 5127788    0    0     0     0  161  278   0  0 100  0  0

 0  0      0 23940396 178804 5127788    0    0     0    32  374  329  1  1 98  0  0

[root@ip-10-128-160-140 ~]#

  

The fist line lists six different categories for which stats will be displayed. The consecutive lines gives all data you need to interpret if you are running into any system level bottlenecks. All the data collected is in "kb" by default.

  Below info in from "man vmstat" and self explanatory which gives details of every column for which stats are collected.




FIELD DESCRIPTION FOR VM MODE

   Procs

       r: The number of processes waiting for run time.

       b: The number of processes in uninterruptible sleep.


   Memory

       swpd: the amount of virtual memory used.

       free: the amount of idle memory.

       buff: the amount of memory used as buffers.

       cache: the amount of memory used as cache.

       inact: the amount of inactive memory. (-a option)

       active: the amount of active memory. (-a option)


   Swap

       si: Amount of memory swapped in from disk (/s).

       so: Amount of memory swapped to disk (/s).


   IO

       bi: Blocks received from a block device (blocks/s).

       bo: Blocks sent to a block device (blocks/s).


   System

       in: The number of interrupts per second, including the clock.

       cs: The number of context switches per second.


   CPU

       These are percentages of total CPU time.

       us: Time spent running non-kernel code. (user time, including nice time)

       sy: Time spent running kernel code. (system time)

       id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.

       wa: Time spent waiting for IO. Prior to Linux 2.5.41, included in idle.

       st: Time stolen from a virtual machine. Prior to Linux 2.6.11, unknown.


  With systems who have larger memory its worth running the command with "-S" option and specifying "M" which would capture stats in "MB" which is more human readable.



[root@ip-10-128-160-140 ~]# vmstat -S M 1 3

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----

 r  b   swpd   free   buff  cache   si   so       bi    bo   in   cs      us sy id wa st

 0  0      0  23386    174   5000    0    0      0     5      3    6       0  0 100  0  0

 0  0      0  23386    174   5000    0    0      0     0     262  299  0  0 100  0  0

 0  0      0  23386    174   5000    0    0      0     0     140  256  0  0 100  0  0

[root@ip-10-128-160-140 ~]#

  
With above info a system administrators can identify system bottlenecks or atleast get a data point on where he needs to dig deeper to get to the RC of the performance problem they are troubleshooting. I plan to write another post with examples where i will manually stimulate system bottlenecks and run vmstat in parallel to get data points which wold point to the problem .