Hourly Logrotate

                                         Hourly Logrotate 

  In earlier post (System Stats) it was recommended to run basic linux tools at regular interval to collect necessary stats to go back and review incase there were any incident and system level details need to be reviewed .
  Issue with regularly collecting this info is obviously we can run out of local disk space. This can be handled if we can use log rotate accurately and only retain details on local FS for lets say few hrs while move older logs to MFS or some other distributed storage.  This Blog shows a way on how to utilize Logrotate to rotate all /opt/mapr/logs/*.out logs collected by system stats and move them to respective files every hr .

1) Create "systemstat" file under "/etc/logrotate.d" location with below configuration.

[root@node107rhel72 logrotate.d]# cat systemstat 

/opt/mapr/logs/*.out {

su  mapr mapr

create 777 mapr mapr

size 10k

daily

rotate 10

compress

delaycompress

copytruncate

} 

Note :-  Make sure all the system stat logs are owned by mapr:mapr (/opt/mapr/logs/*.out)

2) Now since Log rotate can only be run depending on file size or daily , we can use cron.hourly to run the logrotate every hour.

i) Copy logrotate executable under cron.hourly.

cp /etc/cron.daily/logrotate /etc/cron.hourly/

ii) Make sure the file has execute permission for mapr user.

[root@node107rhel72 cron.hourly]# ls -l logrotate 

-rwxr-xr-x. 1 root root 180 Jan  3 19:25 logrotate

iii) Finally modify log rotate file to run "/etc/logrotate.d/systemstat" hourly .

[root@node107rhel72 cron.hourly]# cat logrotate 

#!/bin/sh

/usr/sbin/logrotate -f /etc/logrotate.d/systemstat

EXITVALUE=$?

if [ $EXITVALUE != 0 ]; then

    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"

fi

exit 0

[root@node107rhel72 cron.hourly]# 

Note :- Incase if you see below error in messages file then it could be due to SELinux and might have to be disabled.

logrotate: ALERT exited abnormally with [1]

Verify :

Every hr when the cron runs below logs are logged under "/var/log/cron" which confirms cron is running hourly as expected .

Jan  4 01:01:26 node107rhel72 CROND[8702]: (root) CMD (run-parts /etc/cron.hourly)

Jan  4 01:01:26 node107rhel72 run-parts(/etc/cron.hourly)[8702]: starting 0anacron

Jan  4 01:01:26 node107rhel72 anacron[8711]: Anacron started on 2018-01-04

Jan  4 01:01:26 node107rhel72 anacron[8711]: Normal exit (0 jobs run)

Jan  4 01:01:26 node107rhel72 run-parts(/etc/cron.hourly)[8713]: finished 0anacron

Jan  4 01:01:26 node107rhel72 run-parts(/etc/cron.hourly)[8702]: starting logrotate

Jan  4 01:01:26 node107rhel72 run-parts(/etc/cron.hourly)[8727]: finished logrotate

Jan  4 02:01:01 node107rhel72 CROND[15773]: (root) CMD (run-parts /etc/cron.hourly)

Jan  4 02:01:01 node107rhel72 run-parts(/etc/cron.hourly)[15773]: starting 0anacron

Jan  4 02:01:01 node107rhel72 anacron[15782]: Anacron started on 2018-01-04

Jan  4 02:01:01 node107rhel72 anacron[15782]: Normal exit (0 jobs run)

Jan  4 02:01:01 node107rhel72 run-parts(/etc/cron.hourly)[15784]: finished 0anacron

Jan  4 02:01:01 node107rhel72 run-parts(/etc/cron.hourly)[15773]: starting logrotate

Jan  4 02:01:01 node107rhel72 run-parts(/etc/cron.hourly)[15817]: finished logrotate

Now i am changing date couple of times to make sure the logs are rotated and zipped.

[root@node107rhel72 ~]# date -s "4 Jan 2018 00:00:50"

Thu Jan  4 00:00:50 PST 2018

[root@node107rhel72 ~]# date -s "4 Jan 2018 01:00:50"

Thu Jan  4 01:00:50 PST 2018

[root@node107rhel72 ~]# date -s "4 Jan 2018 02:00:50"

Thu Jan  4 02:00:50 PST 2018

[root@node107rhel72 ~]# ls -ltr /opt/mapr/logs/| grep vmstat.node107rhel72

-rw-r--r--. 1 mapr mapr      4096 Jan  3 23:01 vmstat.node107rhel72.out.4.gz

-rw-r--r--. 1 mapr mapr      8585 Jan  4 00:01 vmstat.node107rhel72.out.3.gz

-rw-r--r--. 1 mapr mapr      8192 Jan  4 01:01 vmstat.node107rhel72.out.2.gz

-rw-r--r--. 1 mapr mapr      4096 Jan  4 02:01 vmstat.node107rhel72.out.1

-rw-r--r--. 1 mapr mapr      4096 Jan  4 02:01 vmstat.node107rhel72.out