Increasing Log retention
Logs are the first place we check when we need to troubleshoot an issue looking for error's or any clue what went wrong. Some times default log retention are not enough for us to go back in time to figure out actually what happened and get RC of the issue. This blogs talk about various configs which can be tweaked for longer log retention for various services.
MFS : In configuration file
mfs.max.logfile.size.in.mb
Lets say we put this value to 10Gb then you will have 5 log file (mfs.log-3 to mfs.log-3.4) of 2 GB each.
CLDB : In configuration file /opt/mapr/conf/log4j.cldb.properties
Lets say i set hadoop.log.maxfilesize to 1024Mb and hadoop.log.maxbackupindexto 10 then I will have 10 GB to total RM logs before they are purged.
Zookeeper : By default zookeeper keeps 10mb of logs and max backup index upto 4
ZK log size is controlled by below log4j.properties /opt/mapr/zookeeper/zookeeper-3.4.5/conf/log4j.properties file
Logs are the first place we check when we need to troubleshoot an issue looking for error's or any clue what went wrong. Some times default log retention are not enough for us to go back in time to figure out actually what happened and get RC of the issue. This blogs talk about various configs which can be tweaked for longer log retention for various services.
MFS : In configuration file
/opt/mapr/conf/mfs.conf increase value for below config.mfs.max.logfile.size.in.mb
maxSizePerLogFile = maxLogSize / MAX_NUM_OF_LOG_FILES
where
maxLogSize =total amount of space that MFS log files can consumeMAX_NUM_OF_LOG_FILES =total number of MFS log files ( 5 hard coded )
Lets say we put this value to 10Gb then you will have 5 log file (mfs.log-3 to mfs.log-3.4) of 2 GB each.
CLDB : In configuration file /opt/mapr/conf/log4j.cldb.properties
log4j.appender.R.MaxFileSize ( Size of each cldb.log before rolling over )
log4j.appender.R.MaxBackupIndex ( Number of cldb.log* before oldest one is deleted )
Lets say i set log4j.appender.R.MaxFileSize to 1024Mb and log4j.appender.R.MaxBackupIndex to 10 then I will have 10 GB to total CLDB logs before they are purged.
Resource Manager : In configuration file /opt/mapr/hadoop/hadoop-<Version>/etc/hadoop/log4j.properties
hadoop.log.maxfilesize ( Size of each yarn-mapr-resourcemanager-<hostname>.log before rolling over )
hadoop.log.maxbackupindex ( Number of yarn-mapr-resourcemanager-<hostname>.log* before oldest one is deleted )
Lets say i set hadoop.log.maxfilesize to 1024Mb and hadoop.log.maxbackupindexto 10 then I will have 10 GB to total RM logs before they are purged.
Zookeeper : By default zookeeper keeps 10mb of logs and max backup index upto 4
ZK log size is controlled by below log4j.properties /opt/mapr/zookeeper/zookeeper-3.4.5/conf/log4j.properties file
# Max log file size of 10MB log4j.appender.ROLLINGFILE.MaxFileSize=10MB # uncomment the next line to limit number of backup files #log4j.appender.ROLLINGFILE.MaxBackupIndex=10 log4j.appender.ROLLINGFILE.MaxBackupIndex=4Ideally to retain logs for longer period set log4j.appender.ROLLINGFILE.MaxFileSize to 512 MB and log4j.appender.ROLLINGFILE.MaxBackupIndex to 10 followed by ZK restart on all ZK nodes in rolling fashion for the new config to be picked.
No comments:
Post a Comment