Jstat

                                                                 

                                              Jstat

A lot of time while troubleshooting performance issue we end up narrowing down to point where Java process went on break for doing Full GC . During this break clients wait on response and obviously there is delay in response, which is seen as performance issue if the java process goes in full GC very often indicating Heap memory given to Java process is not enough and just increasing the heap memory followed by restart of the process should alleviate the problem if there are no memory leak. To find what is accumulated in process heap you can follow below blog.

http://abizeradenwala.blogspot.com/2017/03/heap-dump.html

Example : if CLDB doesn't respond and on reviewing CLDB logs we see below message. It is very likely Java process was doing Full GC hence stopped .

2016-03-02 17:12:30,489 WARN CLDBServer [Health-Monitor]: JVM Monitor thread not scheduled for 2440 milliseconds. This could be due to one of the following: GC pause, all CPUs being busy, or a JVM freeze.

To confirm if GC was kicked in during the time we can use jstat to get details, Here FGC and GCT are most important which will give number of times Java process went for full GC and total number of seconds spent in GC . Obviously we need to take few iteration a minute or 2 apart  during the time of issue and confirm, if the counters increase it would be confirmed GC is kicking in and cause of JVM pause.  

Examples :  jstat -gcutil <Java-PID> <Milli Sec> <Number of iteration>

[root@node9 ~]# jstat -gcutil 9424 1000 10

  S0     S1       E          O      P     YGC     YGCT    FGC    FGCT     GCT   

  0.14   0.00  80.11   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.11   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.13   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.13   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.13   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.13   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.13   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.13   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.18   2.14  99.90    112    0.864     0    0.000    0.864

  0.14   0.00  80.19   2.14  99.90    112    0.864     0    0.000    0.864

[root@node9 ~]# jstat -gc 9424 1000 10

 S0C        S1C       S0U    S1U      EC       EU              OC            OU          PC      PU       YGC     YGCT    FGC    FGCT     GCT   

32448.0 32448.0  45.6   0.0   260032.0 224388.9  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 224405.3  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228188.9  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228342.3  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228342.3  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228353.2  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228353.2  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228369.5  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228369.5  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

32448.0 32448.0  45.6   0.0   260032.0 228369.5  259776.0    5550.0   33536.0 33503.4    112    0.864   0      0.000    0.864

• S0C – S0 Capacity; the capacity in KB of Survivor Space 0.
• S1C – S1 Capacity; the capacity in KB of Survivor Space 1.
• S0   --   Survivor space 0 utilization as a percentage of the space's current capacity.
• S0U – S0 Utilization; how much space in KB is being used in Survivor space 0.
• S1U – S1 Utilization; how much space in KB is being used in Survivor space 1.
• S1 -- Survivor space 1 utilization as a percentage of the space's current capacity.
• EC – Eden Capacity; the capacity, in KB, of Eden.
• EU – Eden Utilization; how much space, in KB, is being used in Eden.
• E --   Eden space utilization as a percentage of the space's current capacity.
• OC – Old Capacity; the capacity, in KB, of the Old Generation.
• OU – Old Utilization; how much space, in KB, is being used by the Old Generation.
• O --  Old space utilization as a percentage of the space's current capacity.
• PC – Perm Capacity; the capacity, in KB, of the Permanent Generation.
• PU – Perm Utilization; how much space, in KB, is being used by the Permanent Generation.
• P -- Permanent space utilization as a percentage of the space's current capacity.
• YGC – Young Garbage Collections; the number of young garbage collections that this JVM has performed so far.
• YGCT – Young Garbage Collection Time; the number of seconds that this JVM has spent performing young garbage collections so far.
• FGC – Full Garbage Collections; the number of full garbage collections this JVM has performed so far.
• FGCT – Full Garbage Collection Time; the number of seconds that this JVM has spent performing young garbage collections so far.
• GCT – Garbage Collection Time; the total number of seconds that this JVM has spent performing garbage collections so far.  (YGCT + FGCT).