Category Archives: HPOM

UXMON: EXT3-fs: checktime reached, running e2fsck is recommended

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : warning
OM Server Time: 2016-08-31 10:22:49
Message : UXMON: EXT3-fs: checktime reached, running e2fsck is recommended
Msg Group : OS
Application : dmsg_mon
Object : EXT3
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Check file /etc/fstab to see if your root filesystem has the number 1 . All the other filesystems should have a number 2

root@linux:~ # cat /etc/fstab
/dev/cciss/c0d0p1 / ext3 acl,user_xattr 1 1
/dev/cciss/c0d0p3 /boot ext3 acl,user_xattr 1 2
/dev/localVG/var /var ext3 acl,user_xattr 1 2
/dev/localVG/app.software /usr/software ext3 acl,user_xattr 1 2
/dev/localVG/dat.users /usr/users ext3 acl,user_xattr 1 2
/dev/localVG/dat.audit /audit ext3 acl,user_xattr 1 2
/dev/cciss/c0d0p2 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0

Create the file /forcefsck and reboot the server if possible

root@linux:~ # ls -l /forcefsck
/bin/ls: /forcefsck: No such file or directory

root@linux:~ # touch /forcefsck

root@linux:~ # shutdown -r now

Run dmesg -c twice to clear the dmesg buffer

root@linux:~ # dmesg -c
root@linux:~ # dmesg -c

UXMON: Volume UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 should be mounted on /srv. Please check your

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : warning
OM Server Time: 2016-07-22 13:31:16
Message : UXMON: Volume UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 should be mounted on /srv. Please check your vfstab fstab or filesystems file. Please also check: UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8
Msg Group : OS
Application : volmon
Object : LVM
Event Type : NONE
Instance Name : NONE
Instruction : No

Seems like a HPOM bug. It’s a Suse Linux 11 SP3 with a btrfs.

root@linux:~ # df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/dm-12 509G 16G 492G 4% /srv

root@linux:~ # blkid | grep dm-12
root@linux:~ #

Running blkid against the device multipath shows in blkid

root@linux:~ # blkid /dev/dm-12
/dev/dm-12: UUID=”c7c47b25-30d8-42bc-8ca8-13f939b5c7b8″ UUID_SUB=”ebe2d68d-0b4f-4586-bd40-6476a824f170″ TYPE=”btrfs”

Create a script to run blkid each 5 minutes

root@linux:~ # vi /root/blkid_srv.sh
#!/bin/bash
# To solve problem UXMON: Volume UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 should be mounted on /srv
/sbin/blkid `mount | grep srv | awk ‘{print $1’}`

Give execute permission to run the script

root@linux:~ # chmod 700 /root/blkid_srv.sh

Install it on crontab

root@linux:~ # crontab -e
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /root/blkid_srv.sh

 

Failure: At least one ITO agent process is not running on linux.setaoffice.com

I was having some problems with opcacta

root@linux:~ # ovc -status
coda OV Performance Core COREXT (23107) Running
opcacta OVO Action Agent AGENT,EA Aborted
opcle OVO Logfile Encapsulator AGENT,EA (23073) Running
opcmona OVO Monitor Agent AGENT,EA (6052) Running
opcmsga OVO Message Agent AGENT,EA (22896) Running
opcmsgi OVO Message Interceptor AGENT,EA (22979) Running
ovbbccb OV Communication Broker CORE (22700) Running
ovcd OV Control CORE (22687) Running
ovconfd OV Config and Deploy COREXT (22725) Running

Looking at /var/opt/OV/log/System.txt after stopping and starting, it shows a message saying that there is a problem with opcacta

root@linux:~ # tail -20 /var/opt/OV/log/System.txt
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:31 2016: opcacta (10806/47113196044032): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:31 2016: ovcd (10437/47099555100992): (ctrl-208) Component ‘opcacta’ with pid 10806 exited with exit value ‘0’. Restarting component.
0: ERR: Fri Apr 29 10:47:36 2016: ovcd (10437/47099477543232): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:36 2016: ovc (10434/47239787368608): (ctrl-7) Error in the target component.
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:36 2016: ovcd (10437/47099573193024): (ctrl-208) Component ‘opcacta’ with pid 12760 exited with exit value ‘0’. Restarting component.
0: INF: Fri Apr 29 10:47:37 2016: coda (10779/47049355707824): SCOPE datasource initialization succeeded
0: ERR: Fri Apr 29 10:47:41 2016: ovcd (10437/47099450292544): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: ERR: Fri Apr 29 10:47:42 2016: ovcd (10437/47099573193024): (ctrl-94) Component ‘opcacta’ exited after very short runtime, therefore it will not be automatically restarted. Use ‘ovc -start opcacta’.

Stop OVC

root@linux:~ # ovc -kill

Stop OVPA

root@linux:~ # ovpa stop

Shutting down Perf Agent collection software
Shutting down scopeux, pid(s) 13146
Waiting on 13146 (10 more tries)
The Perf Agent collector, scopeux has been shut down successfully.
NOTE: The ARM registration daemon ttd will be left running.

Shutting down the alarm generator perfalarm, pid(s) 13244
The perfalarm process has terminated

Remove log* files under /var/opt/perf/datafiles/

root@linux:~ # rm /var/opt/perf/datafiles/log*
rm: remove regular file `/var/opt/perf/datafiles/logappl’? y
rm: remove regular file `/var/opt/perf/datafiles/logdev’? y
rm: remove regular file `/var/opt/perf/datafiles/logglob’? y
rm: remove regular file `/var/opt/perf/datafiles/logindx’? y
rm: remove regular file `/var/opt/perf/datafiles/logpcmd0′? y
rm: remove regular file `/var/opt/perf/datafiles/logproc’? y
rm: remove regular file `/var/opt/perf/datafiles/logtran’? y

Remove *q files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*q
rm: remove regular file `/var/opt/OV/tmp/OpC/aaoorbCvq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/actagtq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpicmaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpimaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtq’? y

Remove *df files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*df
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtdf’? y

Restart HPOM and OVPA

root@linux:~ # /opt/perf/bin/ovpa start

The Perf Agent scope collector is being started.
The ARM registration daemon ttd is already running.
It will be signaled to reprocess its configuration file.

The Performance collection daemon
/opt/perf/bin/scopeux has been started.

The coda daemon /opt/OV/lbin/perf/coda has been started.
It will be fully operational in a few minutes.

The Perf Agent alarm generator is being started.
The alarm generator /opt/perf/bin/perfalarm
has been started.

root@linux:~ # ovc -start

root@linux:~ # ovc -status
coda OV Performance Core COREXT (22782) Running
opcacta OVO Action Agent AGENT,EA (23343) Running
opcle OVO Logfile Encapsulator AGENT,EA (23426) Running
opcmona OVO Monitor Agent AGENT,EA (23501) Running
opcmsga OVO Message Agent AGENT,EA (23238) Running
opcmsgi OVO Message Interceptor AGENT,EA (23459) Running
ovbbccb OV Communication Broker CORE (22755) Running
ovcd OV Control CORE (22747) Running
ovconfd OV Config and Deploy COREXT (22889) Running

UXMON: ServiceGuard: Package(s) are running on ADOPTIVE node(s): infraHP0 (linux02)

UXMON: ServiceGuard: Package(s) are running on ADOPTIVE node(s): infraHP0 (linux02)

Node : linux01.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2016-02-22 14:54:56
Message : UXMON: ServiceGuard: Package(s) are running on ADOPTIVE node(s): infraHP0 (linux02)
Msg Group : OS
Application : sgmon
Object : cmviewcl
Event Type :
not_found

Instance Name :
not_found

Instruction : Check with cmviewcl -v;

If package was down the customer has to be informed about the downtime.

Do not close this case before it is resolved.
As long as this EWM-case is not resolved or closed, monitoring is disabled
In any case inform technical lead!

This ticket is about the cluster package that it is running on alternate node.

root@linux:~ # cp -p /var/opt/OV/bin/instrumentation/sg_mon.cfg /var/opt/OV/conf/OpC/sg_mon.cfg
root@linux:~ # vi /var/opt/OV/conf/OpC/sg_mon.cfg
# PKG[0]=XYZ Package name 1
# PKG_NODE[0]=ABC Primary node on which the package must run
# Define as * will disable running on adoptive node check
# PKG_SWTCH[0]=1 Set to 1 if Package_switching should be ENABLED
# Set to 0 if Package_switching must not be ENABLED

In our case, it was disabled the node check for the package

root@linux:~ # vi /var/opt/OV/conf/OpC/sg_mon.cfg
PKG[0]=infraHP0; PKG_NODE[0]=*; PKG_SWTCH[0]=0

UXMON MODULE got block

Node : solaris.setaoffice.com
Node Type : Sun SPARC (HTTPS)
Severity : warning
OM Server Time: 2015-12-12 23:27:48
Message : UXMON MODULE dfmon got block
Msg Group : OS
Application : uxmon
Object : uxmon
Event Type :
not_found

Instance Name :
not_found

Instruction : This message informs about a uxmon module that found an OS command
that get blocked. This cause that module not able to complete its task
and be pending forever.

Please check that module and why gets blocked.
For this purpose use manual module execution in debug mode:
/var/opt/OV/bin/instrumentation/UXMONbroker -d (case of HTTPS – OVO8 agent)
/var/opt/OV/bin/OpC/cmds/UXMONbroker -d (case of DCE – OVO7 agent)

If needed disable that specific module.

Please, inform the UX Admin or Technical Lead of this box about this situation

A module is running over 5 minutes (default). You may want to increase the timeout or research why the module is running over 5 minutes

root@solaris:/ # cp /var/opt/OV/bin/instrumentation/UXMONbroker.cfg /var/opt/OV/conf/OpC

root@solaris:/ # vi /var/opt/OV/conf/OpC/UXMONbroker.cfg
##########################
# TIMEOUT for OS commands. It will wait only those seconds, when this is timeout, it will trigger ‘got block’ alarm.
$UXMON_OS_TIMEOUT = 1800 ;

DBSPI20-1: The configuration file /var/opt/OV/dbspi/dbtab does not exist. Please configure this node

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2015-12-22 18:30:07
Message : DBSPI20-1: The configuration file /var/opt/OV/dbspi/dbtab does not exist. Please configure this node.
Msg Group : DBSPI
Application : Oracle
Object : dbspicao
Event Type : DBSPI
Instance Name :
not_found

Instruction : DBSPI20-1: The configuration file does not exist. Please configure this node.

Probable Cause: The monitored node has not been configured for DB-SPI, or the node to which
the DB-SPI metric templates were distributed has no databases to monitor.

Suggested Action: Use the DBSPI Config application in the IT/O Application Bank
to configure one or more databases to monitor as explained in the DB-SPI User’s
Guide.

If you do not intend to monitor databases on this system, modify the IT/O template assignment
to exclude the node from DBSPI template distribution.

Troubleshooting Tips in DBMon User Guide [ftp://internalftp.setaoffice.com/DBMON/Documents/DBMon-User_Guide.pdf] may provide more detailed suggestions.

In this case you need to create a configuration file /var/opt/OV/dbspi/dbtab and then set the parameters according to your organization.

You can also remove the monitoring if you don’t have a database on the server

UXMON: Swap space usage exceeded configured threshold 80%

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2015-12-22 16:07:01
Message : UXMON: Swap space usage exceeded configured threshold 80%
Msg Group : OS
Application : swapmon
Object : swap
Event Type :
not_found

Instance Name :
not_found

Instruction : No

This incident is about swap space over 80% threshold. Since I’m not adding more swap space, I increased the threshold

root@linux:~ # vi /var/opt/OV/conf/OpC/swap_mon.cfg
###############################################################################
# swap_mon.cfg #
###############################################################################
# UX_MON EMEA: Monitoring UNIX #
# FILE: @(#)$Header: swap_mon.cfg,v 1.2 2005/03/26 19:11:07 hpscelo Exp $ #
# Released: July 2005 #
# =========================================================================== #
# Copyright (c) 2005 Hewlett Packard – All Rights Reserved. #
###############################################################################
#
# All lines which start with a hash-sign (#) will be ignored
# the whole config file is case-insensitive
#
# THERE ARE NO SYNTAX CHECKS SO MAKE SURE THAT YOUR CONFIGURATION
# IS CORRECT!!!
#
# Every config line has the following layout:
# total [ []]
#
# Every config line must start with “total” because every check is
# performed on the totally free space
#
# The percent used must be given as a positive integer, it must be
# inbetween 0 and 100. If the given percentage is exceeded, an alarm
# is raised.
#
# In the third column you have to configure a severity which is used for
# this alert. Possible severities are: warning, major, critical
#
# With the alert type you can configure which alarm will be raised. Possible
# values are:
# B -> Browser
# N -> Browser+Notification
# T -> Browser+Trouble Ticket
# NT -> Browser+Notification+Trouble Ticket
#
# The from-to gives the time of the day when the script shall run.
# The from and the to time are given as 4-digit-numbers in the 24h format
# So “all day long” would be “0000-2400” (in the configline without the quotes!)
# If you don’t configure a from-to time, the script will run all day long
#
# If you have configured the daytime on which the script shall run, you can also
# configure on which days the script will run. The days are given with crontab
# syntax. You can either give multiple days separated with commas from each other
# (e.g. “1,3,4” -> Monday, Wednesday, Thursday) or you can give span of days
# separated with a hyphen (e.g. “2-4” -> from Tuesday until Thursday).
# In this case the first number has always to be lower than the second one!!!
# As you could already see in the examples, the week starts with “0” for sunday
# and ends with “6” for saturday.
#
# You can configure multiple percent_used entries for the same day(s), the
# script will always raise the alert of the highest threshold that is exceeded
###############################################################################

################################################################################
# Example configuration
###############################################################################
#
#total percent_used severity Alert FROM-TO Days
total 97 major T 0000-2400 *

Another case is that you can have more than one swap device configured. If you have an old version, configure the priority to check the largest swap device

root@linux:~ # swapon -s
Filename Type Size Used Priority
/dev/dm-1 partition 2097148 382516 1
/dev/dm-84 partition 234860540 204904 -1

Check your /etc/fstab file for the swap devices

root@linux:~ # grep swap /etc/fstab
/dev/mapper/vgroot-lv_swap swap swap defaults 0 0
/dev/mapper/vgswap-lv_swap swap swap defaults 0 0

dm-1 is /dev/mapper/vgroot-lv_swap and dm-84 is /dev/mapper/vgswap-lv_swap

root@linux:~ # ls -l /dev/mapper/vgroot-lv_swap
lrwxrwxrwx. 1 root root 7 Apr 20 12:30 /dev/mapper/vgroot-lv_swap -> ../dm-1
root@linux:~ # ls -l /dev/mapper/vgswap-lv_swap
lrwxrwxrwx. 1 root root 8 Feb 18 13:53 /dev/mapper/vgswap-lv_swap -> ../dm-84

In the fourth column, add the property pri to set the priority for the swap devices. I put the largest one as first. The largest one will have a priority number between 0 and 32767

root@linux:~ # grep swap /etc/fstab
/dev/mapper/vgroot-lv_swap swap swap defaults,pri=0 0 0
/dev/mapper/vgswap-lv_swap swap swap defaults,pri=1 0 0

Run /var/opt/OV/bin/instrumentation/UXMONbroker -d swapmon and then check the log file

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -d swapmon
>>Debug mode activated
>>Opened the logfile: /var/opt/OV/log/OpC/swap_mon.log
>>Using Default config file in /var/opt/OV/bin/instrumentation/swap_mon.cfg
>>UXMONswapmon::PARSER_CFG start parsing the swap_mon.cfg…….
>>Open rearmfile /var/opt/OV/log/OpC/swap_mon.rearm.log
>>Module interval setting: 0
>>process info: 60201 /opt/OV/nonOV/perl/a/bin/perl -I/var/opt/OV/bin/instrumentation /var/opt/OV/bin/instrumentation/UXMONswapmon -d -c /var/opt/OV/conf/OpC/swap_mon.cfg -l /var/opt/OV/log/OpC/swap_mon.log
>>configuration file: /var/opt/OV/bin/instrumentation/swap_mon.cfg
>>lock file: /var/opt/OV/tmp/OpC/UXMONswapmon_swap_mon.lock
>>get lock to read /var/opt/OV/tmp/OpC/UXMONswapmon_swap_mon.lock
>>previous process 42318 status: stopped
>>no previous process running
>>run the process now
>>Logged this info…:Wed Apr 27 15:50:50 2016 : INFO : UXMONswapmon is running now, pid=60201
>>Process……
>>UXMONswapmon::GetUsedSwap…….
>>Used Swap is 25
>>configure item total, thres is 95
>>Function: SCHED_Compare with 00:00 15:50
>>Function: SCHED_Compare with 15:50 24:00
>>Function: LogReArms, REARM enable is 0 …
>>ReleaseLock: 60201 1461783050 0 stopped 60201 /opt/OV/nonOV/perl/a/bin/perl -I/var/opt/OV/bin/instrumentation /var/opt/OV/bin/instrumentation/UXMONswapmon -d -c /var/opt/OV/conf/OpC/swap_mon.cfg -l /var/opt/OV/log/OpC/swap_mon.log
>>Logged this info…:Wed Apr 27 15:50:50 2016 : INFO : UXMONswapmon end, pid=60201

root@linux:~ # cat /var/opt/OV/log/OpC/swap_mon.log
Tue Apr 19 08:56:18 2016 : SWAPMON: SWP_001::major::T::Swap space usage ACTUAL=238% reached or exceeded configured threshold (95%)
Tue Apr 19 08:56:18 2016 : INFO : UXMONswapmon end, pid=103729
Tue Apr 19 09:07:17 2016 : INFO : UXMONswapmon is running now, pid=124473

The old version count only one swap device. Deploy the new version so that it will count the total swap memory.

DMESG-UNCLASSIFIED: to probe existing LUN id 0x0 failed with errno of 6.

DMESG-UNCLASSIFIED: to probe existing LUN id 0x0 failed with errno of 6.

Node : hp-ux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : major
OM Server Time: 2015-11-06 10:30:53
Message : DMESG-UNCLASSIFIED: to probe existing LUN id 0x0 failed with errno of 6.
Msg Group : OS
Application : HPUX_dmesg
Object : dmesg_UNCLASSIFIED
Event Type :
not_found

Instance Name :
not_found

Instruction : No

 

Message showing in dmesg and /var/adm/syslog/syslog.log

root@hp-ux:/root # dmesg | grep probe
An attempt to probe existing LUN id 0x0 failed with errno of 22.
An attempt to probe existing LUN id 0x0 failed with errno of 22.

root@hp-ux:/root # grep probe /var/adm/syslog/syslog.log | tail
Nov 4 16:08:31 vlunx012 vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Nov 4 16:08:31 vlunx012 vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Nov 4 16:08:33 vlunx012 vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 22.

Scanning the hardware and it is shown that some tapes are in NO_HW status. Removed these tapes

root@hp-ux:/root # ioscan -fnkNC lunpath | grep NO_HW
lunpath 139 0/0/4/0.0xb.0x0 eslpt NO_HW LUN_PATH LUN path for tape95
lunpath 147 0/0/4/0.0×19.0x0 eslpt NO_HW LUN_PATH LUN path for tape99
lunpath 141 0/0/4/0.0x1a.0x0 eslpt NO_HW LUN_PATH LUN path for tape96
lunpath 144 0/0/4/0.0x1c.0x0 eslpt NO_HW LUN_PATH LUN path for tape97
lunpath 155 0/0/4/0.0x1d.0x0 eslpt NO_HW LUN_PATH LUN path for tape115
lunpath 150 0/0/4/0.0x1e.0x0 eslpt NO_HW LUN_PATH LUN path for tape47
lunpath 157 0/0/4/0.0×24.0x0 eslpt NO_HW LUN_PATH LUN path for tape98
lunpath 408 0/0/6/0.0×26.0x0 eslpt NO_HW LUN_PATH LUN path for tape199
lunpath 410 0/0/6/0.0×27.0x0 eslpt NO_HW LUN_PATH LUN path for tape203
lunpath 412 0/0/6/0.0×28.0x0 eslpt NO_HW LUN_PATH LUN path for tape200
lunpath 415 0/0/6/0.0×29.0x0 eslpt NO_HW LUN_PATH LUN path for tape201
lunpath 418 0/0/6/0.0x2a.0x0 eslpt NO_HW LUN_PATH LUN path for tape212
lunpath 423 0/0/6/0.0x2b.0x0 eslpt NO_HW LUN_PATH LUN path for tape204

root@hp-ux:/root # rmsf -H 0/0/4/0.0xb.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0×19.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1a.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1c.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1d.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1e.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0×24.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×26.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×27.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×28.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×29.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0x2a.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0x2b.0x0

root@hp-ux:/root # ioscan -fnkNC lunpath | grep NO_HW
root@hp-ux:/root #

dmesg output
class : lunpath, instance 139
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 147
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 141
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 144
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 155
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 150
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 157
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 408
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 410
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 412
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 415
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 418
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 423
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 139
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 147
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 141
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 144
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 155
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 150
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 157
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 412
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 408
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 410
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 415
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 418
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 423
An attempt to probe existing LUN id 0x0 failed with errno of 22.

DMESG-UNCLASSIFIED: class :Ttggpatha i stlnse tg

DMESG-UNCLASSIFIED: class :Ttggpatha i stlnse tg

Node : hp-ux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : major
OM Server Time: 2015-11-05 19:07:46
Message : DMESG-UNCLASSIFIED: class :Ttggpatha i stlnse tg
Msg Group : OS
Application : HPUX_dmesg
Object : dmesg_UNCLASSIFIED
Event Type :
not_found

Instance Name :
not_found

Instruction : No

This is a ticket from HPOM dmesg module.
I added string Ttggpatha to /var/opt/OV/conf/OpC/dmsg_mon.cfg

root@hp-ux:~ # cat /var/opt/OV/conf/OpC/dmsg_mon.cfg
no-rewind
tgtpath
tape, instance
option failed
tgtpath
Ttggpatha

DMESG-UNCLASSIFIED: th, instance 175

Node : hp-ux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : major
OM Server Time: 2015-11-06 09:34:49
Message : DMESG-UNCLASSIFIED: th, instance 175
Msg Group : OS
Application : HPUX_dmesg
Object : dmesg_UNCLASSIFIED
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Node : hp-ux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : major
OM Server Time: 2015-11-06 09:52:57
Message : DMESG-UNCLASSIFIED: evice id = loop id, for private loop devices
Msg Group : OS
Application : HPUX_dmesg
Object : dmesg_UNCLASSIFIED
Event Type :
not_found

Instance Name :
not_found

Instruction : No

UXMON: The OFFSET in one peer is greater than the threshold: 400

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : minor
OM Server Time: 2015-10-28 13:01:33
Message : UXMON: The OFFSET in one peer is greater than the threshold: 400
Msg Group : OS
Application : ntpmon
Object : ntp
Event Type :
not_found

Instance Name :
not_found

Instruction : The ntpq -p command shows with one or more peers the
offset in time is greater than the threshold set in the
ntp_mon.cfg.

Please, review the ntp status of your system or increase
the threshold in the ntp_mon if you consider this offset
in time between the clocks of your system and the peer’s clock
is aceptable

Please check /var/opt/OV/log/OpC/ntp_mon.log for more details

Run ntpq -p and then check if the offset is higher than what’s configured on ntp_mon.cfg

root@linux:~ # cat /var/opt/OV/conf/OpC/ntp_mon.cfg
NTP_OFFSET 400
NUM_PEER_FAILS 5

Sync the two ntp servers that you are using and then restart the ntp service on the server

root@linux:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
ntp1 142.40.238.18 6 u 23 64 377 1.761 -23.727 9.886
ntp2 15.179.90.135 5 u 30 64 377 2.003 -48.137 4.418

%d bloggers like this: