Category: Linux

NTP offset is over Critical threshold!

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the

“Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : critical
OM Server Time: 2016-07-25 18:54:18
Message : NTP offset is over Critical threshold!
Msg Group : OS
Application : ntpmon
Object : ntp
Event Type :
not_found

Instance Name :
not_found

Instruction : The ntpq -p command shows with one or more peers the
offset in time is greater than the threshold set in the
ntp_mon.cfg.

Please, review the ntp status of your system or increase
the threshold in the ntp_mon if you consider this offset
in time between the clocks of your system and the peer’s clock
is aceptable

Please check /var/opt/OV/log/OpC/ntp_mon.log for more details

Stopping ntp service

root@linux:~ # service ntp stop
Shutting down network time protocol daemon (NTPD) done

When I tried to run ntpdate in this Suse Linux Enterprise Server 11, I got an error message saying to use sntp

root@linux:~ # ntpdate ntpserver1
!!!!!!!!!!!!!! WARNING !!!!!!!!!!!!!!!!!!

The ntpdate program is deprecated and should not be used any more. To
quote the upstream ntp developers:

“The functionality ntpdate offered is now provided by the ntpd daemon
itself. If you call ntpd with the command line option -q it will
retrieve the current time and set it accordingly.”

Please check the Network Time Protocol (NTP) daemon man page and
http://support.ntp.org/bin/view/Dev/DeprecatingNtpdate
for further information.

You can replace the ntpdate call with “rcntp ntptimeset” to achieve an
inital poll of the servers specified in /etc/ntp.conf.

The program /usr/sbin/sntp offers comparable functionality to ntpdate.
Specifically
sntp -P no -r pool.ntp.org
is equivalent to
ntpdate pool.ntp.org

For further details please refer to the man page of sntp.

root@linux:~ # cat /etc/*release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3

I used my internal NTP server

root@linux:~ # sntp -P no -r ntpserver1

Started NTP server

root@linux:~ # service ntp start
Starting network time protocol daemon (NTPD) done

root@linux:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
ntpserver1 142.40.234.152 3 u 3 64 1 0.665 -27.640 0.001
ntpserver2 142.40.234.152 3 u 2 64 1 1.176 -17.426 0.001
142.40.5.45 .GPS. 1 u 1 64 1 11.257 -13.302 0.001

Check if the server lines in /etc/ntp.conf contain iburst

root@linux:~# grep server /etc/ntp.conf
server 142.40.234.152
server 142.40.234.153
server 10.32.5.29

root@linux:~ # vi /etc/ntp.conf

root@linux:~ # grep server /etc/ntp.conf
server 142.40.234.152 iburst
server 142.40.234.153 iburst
server 10.32.5.29 iburst

I was also having an reoccurence of this problem so I had to set the offset in ntp_mon.cfg

root@linux:~ # ls -l /var/opt/OV/conf/OpC/ntp_mon.cfg
ls: cannot access /var/opt/OV/conf/OpC/ntp_mon.cfg: No such file or directory

root@linux:~ # cp /var/opt/OV/bin/instrumentation/ntp_mon.cfg /var/opt/OV/conf/OpC

root@linux:~ # vi /var/opt/OV/conf/OpC/ntp_mon.cfg
###############################################################################
#@(#) $Id: ntp_mon.cfg 2149 2015-03-03 08:45:34Z zhaofeif $
#@(#) $Rev: 2149 $
#@(#) $Author: zhaofeif $
#@(#) $Date: 2015-03-03 16:45:34 +0800 (Tue, 03 Mar 2015) $
#@(#) $LastChangedBy: zhaofeif $
###############################################################################

############################################
# NTP_MON.CFG
# DESCRIPTION
#——————-
# Monitoring the NTP Daemon
# You can configure the monitoring to watch over
# the offset reported by the ntpq -p command
#
# PARAMS DESCRIPTION
#——————————
# [REARM = TRUE|FALSE]
# If set TRUE (or true), rearm function is enabled, default is disabled
#
# “DISABLE =[yes|no]” : This is the switch for ntpmon running
# NTP_OFFSET_CRITICAL,NTP_OFFSET_WARNING set offset critical and warning threshold for active peer, devaut values are 500 and 250 in ms
# NTP_STRATUM_CRITICAL, NTP_STRATUM_WARNING set offset of stratum of peers which status are one of “*”, “o”, “+” or “#”, the default value are 16 and 10
# NTP_ONLINE_TIME , ntpmon will not check anything if process ntpd is not running more than NTP_ONLINE_TIME(minutes), default value is 10(min)
# ALARM_DELAY, ntpmon allow to delay triggering alarm by setting minutes.
#——————————
# PARAMS REQUIRED
#———————————–
#
# CONFIG FILE
#————————–
############################################
#
# EXAMPLE:
#
#
#################################################################
#
# Set your configuration from here
#REARM = TRUE

DISABLE = NO
NTP_OFFSET_CRITICAL 1500
NTP_OFFSET_WARNING 500
NTP_STRATUM_CRITICAL 15
NTP_STRATUM_WARNING 12
NTP_ONLINE_TIME 3
#ALARM_DELAY 60

# Set your configuration from here

If the last column jitter has a high number, stop NTP service, delete the drift file and start NTP service

root@suselinux:~ # grep drift /etc/ntp.conf
driftfile /var/lib/ntp/drift/ntp.drift # path for drift file

root@redhatlinux:~ # grep drift /etc/ntp.conf
driftfile /var/lib/ntp/drift

root@linux:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
+cator2valedc002 142.40.234.152 3 u 31 128 377 0.465 -70499. 236.299
cator2valedc004 .INIT. 16 u – 256 0 0.000 0.000 0.000
*142.40.5.45 .GPS. 1 u 28 128 377 10.848 -70614. 175.292

Edit file /etc/sysconfig/ntp

root@linux:~ # cat /etc/sysconfig/ntpd
# Drop root to id ‘ntp:ntp’ by default.
OPTIONS=”-x -u ntp:ntp -p /var/run/ntpd.pid -g”

Problems with a NFS share coming from a Linux server and mounted on Linux client

Running df or ls in a NFS share it simply hangs.

See if df or ls hangs. Run strace df -h or strace ls -l and see if it finishes

root@linux:~ # strace df -h

root@linux:~ # strace ls -l

statfs(“/mnt/NFS”,

Umount with -f or -l

-f, –force
Force an unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.)

Note that this option does not guarantee that umount command does not hang. It’s strongly recommended to use absolute paths without symlinks to avoid unwanted readlink and stat syscalls on unreachable NFS in umount.
-l, –lazy
Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore. (Requires kernel 2.4.11 or later.)

su: (to novlwww) root on /dev/console

Jul 21 12:14:48 linux init: Switching to runlevel: 6
Jul 21 12:14:49 linux kernel: [19793592.659965] bootsplash: status on console 0 changed to on
Jul 21 12:15:01 linux syslog-ng[20531]: Log statistics; dropped=’pipe(/dev/xconsole)=0′, dropped=’pipe(/dev/tty10)=0′, processed=’center(queued)=1287386′, processed=’center(received)=1287371′, processed=’destination(messages)=2824′, processed=’destination(mailinfo)=15′, processed=’destination(mailwarn)=0′, processed=’destination(localmessages)=0′, processed=’destination(mailerr)=0′, processed=’destination(newserr)=0′, processed=’destination(netmgm)=0′, processed=’destination(warn)=0′, processed=’destination(console)=0′, processed=’destination(null)=0′, processed=’destination(mail)=15′, processed=’destination(xconsole)=0′, processed=’destination(httpheaders)=63703′, processed=’destination(firewall)=0′, processed=’destination(agsmessages)=1220828′, processed=’destination(acpid)=1′, processed=’destination(soapmessages)=0′, processed=’destination(newscrit)=0′, processed=’destination(newsnotice)=0′, processed=’source(src)=1287371′
Jul 21 12:15:03 linux su: (to novlwww) root on /dev/console
Jul 21 12:15:09 linux sshd[3877]: Received signal 15; terminating.
Jul 21 12:15:09 linux xinetd[5386]: Exiting…
Jul 21 12:15:10 linux auditd[2994]: The audit daemon is exiting.
Jul 21 12:15:10 linux haveged: haveged stopping due to signal 15
Jul 21 12:15:11 linux rpcbind: rpcbind terminating on signal. Restart with “rpcbind -w”
Jul 21 12:15:11 linux kernel: Kernel logging (proc) stopped.
Jul 21 12:15:11 linux kernel: Kernel log daemon terminating.
Jul 21 12:15:11 linux syslog-ng[20531]: Termination requested via signal, terminating;
Jul 21 12:15:11 linux syslog-ng[20531]: syslog-ng shutting down; version=’2.0.9′
Jul 21 12:18:03 linux syslog-ng[2937]: syslog-ng starting up; version=’2.0.9′

This server was rebooted by the user through NAM (Novell Access Manager) Console

Multipathed disk showing error message: Checksum error

Running pvs shows the error message Checksum error and the disk is not part of a volume group

root@linux:~ # pvs | grep NP0_trans_disk_002
/dev/mapper/NP0_trans_disk_002: Checksum error
Couldn’t read volume group metadata.
/dev/mapper/NP0_trans_disk_002: Checksum error
Couldn’t read volume group metadata.
/dev/mapper/NP0_trans_disk_002: Checksum error
Couldn’t read volume group metadata.
/dev/mapper/NP0_trans_disk_002 lvm2 — 50.00g 50.00g

Check all the files inside directory /etc/lvm/backup if there is a file containing the disk that is showing Checksum error

root@linux:~ # cd /etc/lvm/backup
root@linux:~ # grep NP0_trans_disk_002 *
vgNP0trans: device = “/dev/mapper/NP0_trans_disk_002” # Hint only

Restore the volume group configuration

root@linux:~ # vgcfgrestore -f /etc/lvm/backup/vgNP0trans vgNP0trans
/dev/mapper/NP0_trans_disk_002: Checksum error
Couldn’t read volume group metadata.
Restored volume group vgNP0trans

The physical volume starts starts showing volume group information

root@linux:~ # pvs | grep NP0_trans_disk_002
/dev/mapper/NP0_trans_disk_002 vgNP0trans lvm2 a– 46.02g 8.00m

NetIQ Remote Loader

SAP System ID: SMD
Logon Group:
SAP System id: 04
SAP User client number: 100
Logical System Name: SMDCLNT100
Authentication ID: IDM_CPIC
Authentication context: ascssmd.setaoffice.com
Remote Loader connection parameters: hostname=ascssmd.setaoffice.com port=13073 kmo=’sap_dev’

File where some configuration is found

root@linux:/usr/sap/SMD/NOVELL_RemoteLoader # cat config.txt
################################################################################
# Sample configuration file for rdxml or dirxml_jremote. See usage.html for
# descriptions and explanations of these and other available parameters.
#
# This sample demonstrates a configuration for loading the Delimited Text
# driver.
################################################################################
-description “Integration IDM SAP”
-commandport 14033
-connection “address=10.106.136.33 port=13033 handshaketimeout=1000000 keystore=sap_dev.keystore storepass=dirxml”
-trace 10
-tracefile ./trace.log
-tracefilemax 10M
-class com.novell.nds.dirxml.driver.sapumshim.SAPDriverShim

Problems with a NFS share coming from a Linux server and mounted on Linux client

Running df or ls in a NFS share it simply hangs.

See if df or ls hangs. Run strace df -h or strace ls -l and see if it finishes

root@linux:~ # strace df -h

root@linux:~ # strace ls -l

statfs(“/mnt/NFS”,

Try to umount with -f or -l

-f, –force
Force an unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.)

Note that this option does not guarantee that umount command does not hang. It’s strongly recommended to use absolute paths without symlinks to avoid unwanted readlink and stat syscalls on unreachable NFS in umount.

-l, –lazy
Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore. (Requires kernel 2.4.11 or later.)

Using yum behind a proxy on CentOS 6.3 and showing message Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again

I’m behind a proxy and having problems using the EPEL repository on CentOS 6.3.

After installing it, it shows the message Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again

root@linux:~ # yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again

From what I read, yum is not able to access the repository using HTTPS. So replace https for http

root@linux:~ # mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.bak
root@linux:~ # sed ‘s/https/http/g’ /etc/yum.repos.d/epel.repo.bak > /etc/yum.repos.d/epel.repo

Bug 486324 – (yum with proxy) and (repo with https uri) / ssl request doesn’t work throught proxy

UXMON: Volume UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 should be mounted on /srv. Please check your

I’m receiving this ticket

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : warning
OM Server Time: 2016-06-30 17:06:04
Message : UXMON: Volume UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 should be mounted on /srv. Please check your vfstab fstab or filesystems file. Please also check: UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8
Msg Group : OS
Application : volmon
Object : LVM
Event Type : NONE
Instance Name : NONE
Instruction : No

Checking the UXMONbroker I see that it shows that UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 is not mounted.

This server is a SUSE Linux 11 SP3

root@linux:~ # cat /etc/*release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 3
Build no: 1565
Build date: Fri Aug 14 07:53:12 CEST 2015
Kiwi version: 7.02.58
LSB_VERSION=”core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64″

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check volmon
Mon Jul 4 11:52:53 2016 : INFO : UXMONvolmon is running now, pid=33366
connect() failed on local socket: No such file or directory
Internal cluster locking initialisation failed.
WARNING: Falling back to local file-based locking.
Volume Groups with the clustered attribute will be inaccessible.
Finding all volume groups
Finding volume group “vg_log1_dp_14”
Finding volume group “vg_data1_dp_11”
Finding volume group “vg_log1_14”
Finding volume group “vg_log1_dp_12”
Finding volume group “vg_log1_dp_11”
Finding volume group “vg_data1_dp_13”
Finding volume group “vg_data1_dp_14”
Finding volume group “vg_data1_dp_12”
Finding volume group “vg_data1_11”
Finding volume group “vg_log1_dp_13”
Finding volume group “vg_data1_14”
Finding volume group “vg_log1_11”
Finding volume group “vg_data1_12”
Finding volume group “vg_log1_12”
Finding volume group “vg_log1_13”
Finding volume group “vg_data1_13”
Mon Jul 4 11:52:58 2016 : VOLMON: CMA(NONE,NONE) Volume UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 should be mounted on /srv .Please also check: UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8 UUID=c7c47b25-30d8-42bc-8ca8-13f939b5c7b8
mv: `/dev/null’ and `/dev/null’ are the same file
Mon Jul 4 11:52:58 2016 : INFO : UXMONvolmon end, pid=33366

The filesystem is mounted

root@linux:~ # df -h /srv
Filesystem Size Used Avail Use% Mounted on
/dev/dm-16 509G 14G 494G 3% /srv

This server is using btrfs

root@linux:~ # blkid /dev/dm-16
/dev/dm-16: UUID=”c7c47b25-30d8-42bc-8ca8-13f939b5c7b8″ UUID_SUB=”ebe2d68d-0b4f-4586-bd40-6476a824f170″ TYPE=”btrfs”

Red Hat Linux 5: qla2xxx 0000:05:00.0: scsi(3:4:0): Abort command issued — 1 40e 2002

The server had two single port HBA cards that were receiving a lot of reset commands

root@linux:~ # lspci | grep -i fibre
05:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
08:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
0b:00.0 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
0b:00.1 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)

Jun 12 04:11:47 linnux kernel: qla2xxx 0000:08:00.0: Performing ISP error recovery – ha= ffff8102abac04f8.
Jun 12 04:11:47 linnux kernel: qla2xxx 0000:05:00.0: scsi(3:4:0): Abort command issued — 1 40e 2002.
Jun 12 04:11:47 linnux kernel: qla2xxx 0000:05:00.0: scsi(3:4:0): LOOP RESET ISSUED.
Jun 12 04:11:48 linnux kernel: qla2xxx 0000:05:00.0: qla2xxx_eh_bus_reset: reset succeeded
Jun 12 04:11:48 linnux kernel: qla2xxx 0000:08:00.0: LIP reset occured (f700).
Jun 12 04:11:48 linnux kernel: qla2xxx 0000:08:00.0: LIP occured (f700).
Jun 12 04:11:48 linnux kernel: qla2xxx 0000:08:00.0: LIP reset occured (f7f7).
Jun 12 04:11:48 linnux kernel: qla2xxx 0000:08:00.0: LOOP UP detected (4 Gbps).
Jun 12 04:11:50 linnux kernel: qla2xxx 0000:08:00.0: qla2xxx_eh_host_reset: reset succeeded
Jun 12 04:11:54 linnux kernel: lpfc 0000:0b:00.1: 1:(0):0713 SCSI layer issued Device Reset (11, 0) return x2002
Jun 12 04:12:01 linnux kernel: scsi 1:0:5:0: scsi: Device offlined – not ready after error recovery
Jun 12 04:12:01 linnux kernel: scsi 1:0:5:0: timing out command, waited 22s

We had opened a ticket with Red Hat and they said to check the hardware.

The BUR team was asked to check the tape library and storage team was asked to check the SAN switch. No problems were found.

So we logged a ticket with HP to replace the system board. Two different HBA cards in different PCI slots were having problems.

Replaced system board and riser card.

Jun 16 12:02:49 linnux kernel: lpfc 0000:0b:00.0: 0:(0):0713 SCSI layer issued Device Reset (3, 0) return x2002
Jun 16 12:02:59 linnux kernel: lpfc 0000:0b:00.0: 0:(0):0714 SCSI layer issued Bus Reset Data: x2002
Jun 16 12:03:20 linnux kernel: lpfc 0000:0b:00.0: 0:3172 SCSI layer issued Host Reset Data: x2002
Jun 16 12:03:20 linnux kernel: lpfc 0000:0b:00.0: 0:1303 Link Up Event x1 received Data: x1 xf7 x10 x9 x0 x0 0
Jun 16 12:03:40 linnux kernel: scsi 1:0:3:0: scsi: Device offlined – not ready after error recovery
Jun 16 12:03:40 linnux kernel: scsi 1:0:3:0: timing out command, waited 22s
Jun 16 12:03:55 linnux kernel: lpfc 0000:0b:00.0: 0:(0):0713 SCSI layer issued Device Reset (4, 0) return x2002

But the problem still persisted.

Some parameters in the SAN switch were changed and the problem was solved.

Clustered Linux server showing device-mapper: multipath: Failing path in /var/log/messages

I have a disk presented to 4 servers.

Everyday we receive a notification saying that a specific multipathed disk lost all paths.

The disk is showing a SCSI reservation conflict – SCSI persistent reservations provide the capability to control the access of each node to shared storage devices

May 11 13:35:04 linux kernel: sd 0:0:0:38: reservation conflict
May 11 13:35:04 linux kernel: sd 0:0:0:38: [sdag] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 0:0:0:38: [sdag] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00

May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdag, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 66:0. <————————sdag
May 11 13:35:04 linux kernel: sd 1:0:0:38: reservation conflict
May 11 13:35:04 linux kernel: sd 1:0:0:38: [sdeh] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 1:0:0:38: [sdeh] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdeh, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 128:144. <———————-sdeh
May 11 13:35:04 linux multipathd: 66:0: mark as failed
May 11 13:35:04 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 3

May 11 13:35:04 linux kernel: sd 0:0:1:38: reservation conflict
May 11 13:35:04 linux kernel: sd 0:0:1:38: [sdcc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 0:0:1:38: [sdcc] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdcc, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 69:0.<—————————sdcc
May 11 13:35:04 linux kernel: sd 1:0:1:38: reservation conflict
May 11 13:35:04 linux kernel: sd 1:0:1:38: [sdgg] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 1:0:1:38: [sdgg] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdgg, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 131:192.<————————-sdgg
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev dm-209, sector 5200

May 11 13:35:05 linux multipathd: 128:144: mark as failed
May 11 13:35:05 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 2
May 11 13:35:05 linux multipathd: 69:0: mark as failed
May 11 13:35:05 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 1
May 11 13:35:05 linux multipathd: 131:192: mark as failed

PP0_oraarch_disk_001 (350002ad05071374b) dm-209 3PARdata,VV
size=300G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 0:0:0:38 sdag 66:0 active undef running
|- 1:0:0:38 sdeh 128:144 active undef running
|- 0:0:1:38 sdcc 69:0 active undef running
`- 1:0:1:38 sdgg 131:192 active undef running

Since this is opening support tickets, I removed monitoring for this disk