Core Dumps

                                    Core Dumps

In most Linux systems core dump files are generated after an uncaught signal in a process (as a SIGSEGV or SIGQUIT). Core is state of the process at particular instance and is very helpful for debugging issues.

File where core generation limit is specified at user level.

cat /etc/security/limits.conf | grep core

#        - core - limits the core file size (KB)

#*               soft    core            0

mapr - core unlimited

Current core limit for user "MapR"

[mapr@node9 root]$ ulimit -c

unlimited

[mapr@node9 root]$ ulimit -a

core file size          (blocks, -c) unlimited

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 30

file size               (blocks, -f) unlimited

pending signals                 (-i) 95148

max locked memory       (kbytes, -l) unlimited

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) unlimited

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

The Core Pattern in Kernel :

The core location and pattern is determined by the value stored in "/proc/sys/kernel/core_pattern"

[root@node9 ~]# cat /proc/sys/kernel/core_pattern

/opt/cores/%e.core.%p.%h

[root@node9 ~]#

 This pattern shows how the core file will be generated and where. Two things that can be understood from the output: 1) The filename of the core dump file generated will be “core” and Binary name in the start    2) the current directory used to store it 

You can use the following pattern elements in the core_pattern file:

%p: pid
%: '%' is dropped
%%: output one '%'
%u: uid
%g: gid
%s: signal number
%t: UNIX time of dump
%h: hostname
%e: executable filename
%: both are dropped

 You can detect which node generated the core file (with the hostname), which program generated it (with executable filename ), and also when did it happen (with the unix time).

Below command can be used to re-direct core to customer location.

echo "/opt/cores/%e.core.%p.%h" > /proc/sys/kernel/core_pattern

Configure core dump forever

The changes done above are only applicable until the next reboot. In order to make the change persist reboots, we will need to add the following in “/etc/sysctl.conf“:

i)  tail -2 /etc/sysctl.conf 

# Custom core file pattern and location.

kernel.core_pattern=/opt/cores/%e.core.%p.%h

Now to Load in sysctl settings from the file /etc/sysctl.con

ii) sysctl -p

iii) sysctl -a | grep kernel.core_pattern

kernel.core_pattern = /opt/cores/%e.core.%p.%h (sysctl.conf is the file controlling every configuration under /proc/sys)

Test Core generated in correct Location:

Below command will trigger a segmentation fault in current shell and generate a core file in location we are looking to collect core dump.

[mapr@node9 root]$ kill -s SIGSEGV $$

Segmentation fault (core dumped)

[root@node9 ~]# ls /opt/cores/

bash.core.25469.node9.maprlab.local

[root@node9 ~]#