Advertisements

UXMON: Swap space usage exceeded configured threshold 80%

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2015-12-22 16:07:01
Message : UXMON: Swap space usage exceeded configured threshold 80%
Msg Group : OS
Application : swapmon
Object : swap
Event Type :
not_found

Instance Name :
not_found

Instruction : No

This incident is about swap space over 80% threshold. Since I’m not adding more swap space, I increased the threshold

root@linux:~ # vi /var/opt/OV/conf/OpC/swap_mon.cfg
###############################################################################
# swap_mon.cfg #
###############################################################################
# UX_MON EMEA: Monitoring UNIX #
# FILE: @(#)$Header: swap_mon.cfg,v 1.2 2005/03/26 19:11:07 hpscelo Exp $ #
# Released: July 2005 #
# =========================================================================== #
# Copyright (c) 2005 Hewlett Packard – All Rights Reserved. #
###############################################################################
#
# All lines which start with a hash-sign (#) will be ignored
# the whole config file is case-insensitive
#
# THERE ARE NO SYNTAX CHECKS SO MAKE SURE THAT YOUR CONFIGURATION
# IS CORRECT!!!
#
# Every config line has the following layout:
# total [ []]
#
# Every config line must start with “total” because every check is
# performed on the totally free space
#
# The percent used must be given as a positive integer, it must be
# inbetween 0 and 100. If the given percentage is exceeded, an alarm
# is raised.
#
# In the third column you have to configure a severity which is used for
# this alert. Possible severities are: warning, major, critical
#
# With the alert type you can configure which alarm will be raised. Possible
# values are:
# B -> Browser
# N -> Browser+Notification
# T -> Browser+Trouble Ticket
# NT -> Browser+Notification+Trouble Ticket
#
# The from-to gives the time of the day when the script shall run.
# The from and the to time are given as 4-digit-numbers in the 24h format
# So “all day long” would be “0000-2400” (in the configline without the quotes!)
# If you don’t configure a from-to time, the script will run all day long
#
# If you have configured the daytime on which the script shall run, you can also
# configure on which days the script will run. The days are given with crontab
# syntax. You can either give multiple days separated with commas from each other
# (e.g. “1,3,4” -> Monday, Wednesday, Thursday) or you can give span of days
# separated with a hyphen (e.g. “2-4” -> from Tuesday until Thursday).
# In this case the first number has always to be lower than the second one!!!
# As you could already see in the examples, the week starts with “0” for sunday
# and ends with “6” for saturday.
#
# You can configure multiple percent_used entries for the same day(s), the
# script will always raise the alert of the highest threshold that is exceeded
###############################################################################

################################################################################
# Example configuration
###############################################################################
#
#total percent_used severity Alert FROM-TO Days
total 97 major T 0000-2400 *

Another case is that you can have more than one swap device configured. If you have an old version, configure the priority to check the largest swap device

root@linux:~ # swapon -s
Filename Type Size Used Priority
/dev/dm-1 partition 2097148 382516 1
/dev/dm-84 partition 234860540 204904 -1

Check your /etc/fstab file for the swap devices

root@linux:~ # grep swap /etc/fstab
/dev/mapper/vgroot-lv_swap swap swap defaults 0 0
/dev/mapper/vgswap-lv_swap swap swap defaults 0 0

dm-1 is /dev/mapper/vgroot-lv_swap and dm-84 is /dev/mapper/vgswap-lv_swap

root@linux:~ # ls -l /dev/mapper/vgroot-lv_swap
lrwxrwxrwx. 1 root root 7 Apr 20 12:30 /dev/mapper/vgroot-lv_swap -> ../dm-1
root@linux:~ # ls -l /dev/mapper/vgswap-lv_swap
lrwxrwxrwx. 1 root root 8 Feb 18 13:53 /dev/mapper/vgswap-lv_swap -> ../dm-84

In the fourth column, add the property pri to set the priority for the swap devices. I put the largest one as first. The largest one will have a priority number between 0 and 32767

root@linux:~ # grep swap /etc/fstab
/dev/mapper/vgroot-lv_swap swap swap defaults,pri=0 0 0
/dev/mapper/vgswap-lv_swap swap swap defaults,pri=1 0 0

Run /var/opt/OV/bin/instrumentation/UXMONbroker -d swapmon and then check the log file

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -d swapmon
>>Debug mode activated
>>Opened the logfile: /var/opt/OV/log/OpC/swap_mon.log
>>Using Default config file in /var/opt/OV/bin/instrumentation/swap_mon.cfg
>>UXMONswapmon::PARSER_CFG start parsing the swap_mon.cfg…….
>>Open rearmfile /var/opt/OV/log/OpC/swap_mon.rearm.log
>>Module interval setting: 0
>>process info: 60201 /opt/OV/nonOV/perl/a/bin/perl -I/var/opt/OV/bin/instrumentation /var/opt/OV/bin/instrumentation/UXMONswapmon -d -c /var/opt/OV/conf/OpC/swap_mon.cfg -l /var/opt/OV/log/OpC/swap_mon.log
>>configuration file: /var/opt/OV/bin/instrumentation/swap_mon.cfg
>>lock file: /var/opt/OV/tmp/OpC/UXMONswapmon_swap_mon.lock
>>get lock to read /var/opt/OV/tmp/OpC/UXMONswapmon_swap_mon.lock
>>previous process 42318 status: stopped
>>no previous process running
>>run the process now
>>Logged this info…:Wed Apr 27 15:50:50 2016 : INFO : UXMONswapmon is running now, pid=60201
>>Process……
>>UXMONswapmon::GetUsedSwap…….
>>Used Swap is 25
>>configure item total, thres is 95
>>Function: SCHED_Compare with 00:00 15:50
>>Function: SCHED_Compare with 15:50 24:00
>>Function: LogReArms, REARM enable is 0 …
>>ReleaseLock: 60201 1461783050 0 stopped 60201 /opt/OV/nonOV/perl/a/bin/perl -I/var/opt/OV/bin/instrumentation /var/opt/OV/bin/instrumentation/UXMONswapmon -d -c /var/opt/OV/conf/OpC/swap_mon.cfg -l /var/opt/OV/log/OpC/swap_mon.log
>>Logged this info…:Wed Apr 27 15:50:50 2016 : INFO : UXMONswapmon end, pid=60201

root@linux:~ # cat /var/opt/OV/log/OpC/swap_mon.log
Tue Apr 19 08:56:18 2016 : SWAPMON: SWP_001::major::T::Swap space usage ACTUAL=238% reached or exceeded configured threshold (95%)
Tue Apr 19 08:56:18 2016 : INFO : UXMONswapmon end, pid=103729
Tue Apr 19 09:07:17 2016 : INFO : UXMONswapmon is running now, pid=124473

The old version count only one swap device. Deploy the new version so that it will count the total swap memory.

Advertisements
%d bloggers like this: