Tag: multipath

UXMON: mpatho – Only one FC Switch detected, no SAN FC Switch redundancy

After receiving the information that the error was caused by storage node reboot, checking if HPOM still show the error

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Thu May 14 10:34:20 2020 : INFO : UXMONmpmon is running now, pid=31809
Thu May 14 10:34:20 2020 : Minor: mpatho – Only one FC Switch detected, no SAN FC Switch redundancy
mv: `/dev/null’ and `/dev/null’ are the same file
Thu May 14 10:34:20 2020 : INFO : UXMONmpmon end, pid=31809

Checking multipath for mpatho

root@linux:~ # multipath -ll mpatho
mpatho (350002ac666af374b) dm-74 3PARdata,VV
size=150G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 2:0:0:5 sday 67:32 active ready running
`- 4:0:0:5 sdax 67:16 active ready running

Rescaning server

root@linux:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | bash

Checking multipath for mpatho

root@linux:~ # multipath -ll mpatho
mpatho (350002ac666af374b) dm-74 3PARdata,VV
size=150G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 2:0:0:5 sday 67:32 active ready running
|- 4:0:0:5 sdax 67:16 active ready running
|- 1:0:0:5 sdaz 67:48 active ready running
`- 3:0:0:5 sdba 67:64 active ready running

Checking if HPOM still show the error

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Thu May 14 10:36:26 2020 : INFO : UXMONmpmon is running now, pid=37696
mv: `/dev/null’ and `/dev/null’ are the same file
Thu May 14 10:36:26 2020 : INFO : UXMONmpmon end, pid=37696

Linux LVM: Device /dev/mapper/ECP_data_disk_001 has size of 1048576000 sectors which is smaller than corresponding PV size of 1174372352 sectors. Was device resized?

Running LVM command and it shows an unrelated disk that is complaining it has a unexpected size.

This problem was happening because it is a cluster node and LUN was resized.

On the node that showed the message, I didn’t run any commands to resize it

root@linux02:~ # pvs | grep PP0_data_disk_001
Device /dev/mapper/ECP_data_disk_001 has size of 1048576000 sectors which is smaller than corresponding PV size of 1174372352 sectors. Was device resized?

Run the commands to resize the LUN on the node

root@linux02:~ # multipath -ll ECP_data_disk_001 | grep ready | awk ‘{print “echo 1 > /sys/block/”$3″/device/rescan”‘} | bash
root@linux02:~ # multipathd -k’resize map ECP_data_disk_001′
ok
root@linux02:~ # pvresize /dev/mapper/ECP_data_disk_001
Physical volume “/dev/mapper/ECP_data_disk_001” changed
1 physical volume(s) resized / 0 physical volume(s) not resized

UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy

Also see:
UXMON: mpathb – Only one path detected, no path redundancy
UXMON: volumegroup – Only one path detected, no path redundancy

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2017-12-11 03:28:31
Message : UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy
Msg Group : OS
Application : mpmon
Object : mp
Event Type :
not_found

Instance Name :
not_found

Instruction : The multipathd -k”show map $device topology” command shows more details

Please check /var/opt/OV/log/OpC/mp_mon.log for more details
EventDataSource :

When running a LVM command it is showing that several PVs are duplicate.

root@linux:~ # pvs
Found duplicate PV s3zEE42awIhydJ05hfUzJulPsN8WS266: using /dev/mapper/SY1_disknew_001 not /dev/sdbm
Using duplicate PV /dev/mapper/SY1_disknew_001 from subsystem DM, replacing /dev/sdbm
Found duplicate PV zTZlnwYgW69xzUcTi0riu5euTPiWWnRs: using /dev/mapper/swap_disk_001p1 not /dev/sdbo1
Using duplicate PV /dev/mapper/swap_disk_001p1 from subsystem DM, ignoring /dev/sdbo1
Found duplicate PV hnuWyziFeXRhJ71YpLJX1cLAjOvgtl0q: using /dev/mapper/SCR_DATA_disk_001p1 not /dev/sdbn1
Using duplicate PV /dev/mapper/SCR_DATA_disk_001p1 from subsystem DM, replacing /dev/sdbn1
Found duplicate PV EeqL0LDdMohfnagni16NlRUig3eugbap: using /dev/mapper/SY1_log2_disk_001p1 not /dev/sdbp1
Using duplicate PV /dev/mapper/SY1_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbp1
Found duplicate PV sMAYQABFedTStD589d5ZcDu1ZNtXfXyh: using /dev/mapper/SY1_log1_disk_001p1 not /dev/sdbq1
Using duplicate PV /dev/mapper/SY1_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbq1
Found duplicate PV qFBDDPa7FtN7F97tpUDeKv0cBO94WBr3: using /dev/mapper/SCR_disk_002 not /dev/sdbr
Using duplicate PV /dev/mapper/SCR_disk_002 from subsystem DM, ignoring /dev/sdbr
Found duplicate PV RZvwCnQn48G2A9IifNOLZT9l9ZYdE7yu: using /dev/mapper/SY1_arch_disk_001p1 not /dev/sdbs1
Using duplicate PV /dev/mapper/SY1_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbs1
Found duplicate PV qWPh0v3E1ADowxUuuC8kP8BFJ1bwBsmL: using /dev/mapper/SY1_disk_001p1 not /dev/sdbt1
Using duplicate PV /dev/mapper/SY1_disk_001p1 from subsystem DM, ignoring /dev/sdbt1
Found duplicate PV hi5F4g9FPojaD9J7KH5vHCzRKgd4jQfJ: using /dev/mapper/MR2_log2_disk_001p1 not /dev/sdbu1
Using duplicate PV /dev/mapper/MR2_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbu1
Found duplicate PV KA1AkbCOgfHFGuKazmHAacwqBNPmEyV8: using /dev/mapper/MR2_log1_disk_001p1 not /dev/sdbv1
Using duplicate PV /dev/mapper/MR2_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbv1
Found duplicate PV dy0YaJoSrlZdTM9isfY1QGS6kWBTzs6i: using /dev/mapper/MR2_data_disk_001p1 not /dev/sdbw1
Using duplicate PV /dev/mapper/MR2_data_disk_001p1 from subsystem DM, ignoring /dev/sdbw1
Found duplicate PV khSEnsJ9CC4epw0uOAsaFwlbIKS32Qyj: using /dev/mapper/MR2_arch_disk_001p1 not /dev/sdbx1
Using duplicate PV /dev/mapper/MR2_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbx1
Found duplicate PV PmTTDGWvEvDM8AMUHvqcoRJeA9g4r78D: using /dev/mapper/MRC_log2_disk_001p1 not /dev/sdbi1
Using duplicate PV /dev/mapper/MRC_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbi1
Found duplicate PV ffv1TgDxRIEIyhRoEh46fMGeFuJ1tVtv: using /dev/mapper/MR2_disk_001p1 not /dev/sdby1
Using duplicate PV /dev/mapper/MR2_disk_001p1 from subsystem DM, ignoring /dev/sdby1
Found duplicate PV tCeMdOB1dFuJomhgB061M7MeZNEwptk3: using /dev/mapper/MRC_log1_disk_001p1 not /dev/sdbj1
Using duplicate PV /dev/mapper/MRC_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbj1
Found duplicate PV inuuS3f39VjSuh6q7r5pEZSjzRgRhi5e: using /dev/mapper/sap_disk_001p1 not /dev/sdbz1
Using duplicate PV /dev/mapper/sap_disk_001p1 from subsystem DM, ignoring /dev/sdbz1
Found duplicate PV Ea4CVYCMjYc3eE129uYlUlZKdzhmYNpQ: using /dev/mapper/MRC_data_disk_001p1 not /dev/sdbk1
Using duplicate PV /dev/mapper/MRC_data_disk_001p1 from subsystem DM, ignoring /dev/sdbk1
Found duplicate PV Lx5viH3geNo0zSO07sHcbNlJ9nVmcGPA: using /dev/mapper/MRC_arch_disk_001p1 not /dev/sdbl1
Using duplicate PV /dev/mapper/MRC_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbl1
PV VG Fmt Attr PSize PFree
/dev/mapper/MR2_arch_disk_001p1 vgMR2oraarch lvm2 a–u 19.98g 0
/dev/mapper/MR2_data_disk_001p1 vgMR2data lvm2 a–u 299.98g 99.98g
/dev/mapper/MR2_disk_001p1 vgMR2 lvm2 a–u 99.98g 20.87g
/dev/mapper/MR2_log1_disk_001p1 vgMR2log1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MR2_log2_disk_001p1 vgMR2log2 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_arch_disk_001p1 vgMRCoraarch lvm2 a–u 19.98g 0
/dev/mapper/MRC_data_disk_001p1 vgMRCdata lvm2 a–u 299.98g 39.98g
/dev/mapper/MRC_disk_002 vgMRC lvm2 a–u 149.98g 31.88g
/dev/mapper/MRC_log1_disk_001p1 vgMRClog1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_log2_disk_001p1 vgMRClog2 lvm2 a–u 9.98g 7.98g
/dev/mapper/SCR_ARCH_disk_001p1 vgSCRarch lvm2 a–u 29.96g 4.57g
/dev/mapper/SCR_DATA_disk_001p1 vgSCRdata lvm2 a–u 109.96g 8.40g
/dev/mapper/SCR_LOG1_disk_001p1 vgSCRlog1 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_LOG2_disk_001p1 vgSCRlog2 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_disk_001p1 vgSCR lvm2 a–u 49.96g 3.96g
/dev/mapper/SCR_disk_002 vgSCR lvm2 a–u 49.98g 34.98g
/dev/mapper/SY1_arch_disk_001p1 vgSY1oraarch lvm2 a–u 149.98g 0
/dev/mapper/SY1_data_disk_002 vgSY1data lvm2 a–u 2.64t 0
/dev/mapper/SY1_disk_001p1 vgSY1 lvm2 a–u 49.98g 10.52g
/dev/mapper/SY1_disknew_001 vgSY1 lvm2 a–u 59.98g 44.98g
/dev/mapper/SY1_interf_disk_001 vgSY1interface lvm2 a–u 49.98g 1008.00m
/dev/mapper/SY1_log1_disk_001p1 vgSY1log1 lvm2 a–u 19.98g 9.98g
/dev/mapper/SY1_log2_disk_001p1 vgSY1log2 lvm2 a–u 19.98g 9.98g
/dev/mapper/sap_disk_001p1 vgSAPlocal lvm2 a–u 49.98g 4.79g
/dev/mapper/swap_disk_001p1 vgswap lvm2 a–u 223.98g 0
/dev/sda2 vgroot lvm2 a–u 278.86g 198.27g

Removing disk paths that are duplicate

echo 1 > /sys/block/sdbm/device/delete
echo 1 > /sys/block/sdbo/device/delete
echo 1 > /sys/block/sdbn/device/delete
echo 1 > /sys/block/sdbp/device/delete
echo 1 > /sys/block/sdbq/device/delete
echo 1 > /sys/block/sdbr/device/delete
echo 1 > /sys/block/sdbs/device/delete
echo 1 > /sys/block/sdbt/device/delete
echo 1 > /sys/block/sdbu/device/delete
echo 1 > /sys/block/sdbv/device/delete
echo 1 > /sys/block/sdbw/device/delete
echo 1 > /sys/block/sdbx/device/delete
echo 1 > /sys/block/sdbi/device/delete
echo 1 > /sys/block/sdby/device/delete
echo 1 > /sys/block/sdbj/device/delete
echo 1 > /sys/block/sdbz/device/delete

Checking status

root@linux:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/MR2_arch_disk_001p1 vgMR2oraarch lvm2 a–u 19.98g 0
/dev/mapper/MR2_data_disk_001p1 vgMR2data lvm2 a–u 299.98g 99.98g
/dev/mapper/MR2_disk_001p1 vgMR2 lvm2 a–u 99.98g 20.87g
/dev/mapper/MR2_log1_disk_001p1 vgMR2log1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MR2_log2_disk_001p1 vgMR2log2 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_arch_disk_001p1 vgMRCoraarch lvm2 a–u 19.98g 0
/dev/mapper/MRC_data_disk_001p1 vgMRCdata lvm2 a–u 299.98g 39.98g
/dev/mapper/MRC_disk_002 vgMRC lvm2 a–u 149.98g 31.88g
/dev/mapper/MRC_log1_disk_001p1 vgMRClog1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_log2_disk_001p1 vgMRClog2 lvm2 a–u 9.98g 7.98g
/dev/mapper/SCR_ARCH_disk_001p1 vgSCRarch lvm2 a–u 29.96g 4.57g
/dev/mapper/SCR_DATA_disk_001p1 vgSCRdata lvm2 a–u 109.96g 8.40g
/dev/mapper/SCR_LOG1_disk_001p1 vgSCRlog1 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_LOG2_disk_001p1 vgSCRlog2 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_disk_001p1 vgSCR lvm2 a–u 49.96g 3.96g
/dev/mapper/SCR_disk_002 vgSCR lvm2 a–u 49.98g 34.98g
/dev/mapper/SY1_arch_disk_001p1 vgSY1oraarch lvm2 a–u 149.98g 0
/dev/mapper/SY1_data_disk_002 vgSY1data lvm2 a–u 2.64t 0
/dev/mapper/SY1_disk_001p1 vgSY1 lvm2 a–u 49.98g 10.52g
/dev/mapper/SY1_disknew_001 vgSY1 lvm2 a–u 59.98g 44.98g
/dev/mapper/SY1_interf_disk_001 vgSY1interface lvm2 a–u 49.98g 1008.00m
/dev/mapper/SY1_log1_disk_001p1 vgSY1log1 lvm2 a–u 19.98g 9.98g
/dev/mapper/SY1_log2_disk_001p1 vgSY1log2 lvm2 a–u 19.98g 9.98g
/dev/mapper/sap_disk_001p1 vgSAPlocal lvm2 a–u 49.98g 4.79g
/dev/mapper/swap_disk_001p1 vgswap lvm2 a–u 223.98g 0
/dev/sda2 vgroot lvm2 a–u 278.86g 198.27g

Checking multipath devices. Some devices have only one path

root@linux:~ # multipath -ll
SCR_disk_001 (350002acb5a22374a) dm-26 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:21 sdy 65:128 active ready running
`- 1:0:0:21 sdcb 68:240 active ready running
SY1_log2_disk_001 (350002ac17cd5374a) dm-6 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:6 sdi 8:128 active ready running
MRC_arch_disk_001 (350002ac15504374a) dm-3 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:3 sdf 8:80 active ready running
MR2_data_disk_001 (350002ac17cd8374a) dm-14 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:13 sdp 8:240 active ready running
MR2_disk_001 (350002ac17cd6374a) dm-18 3PARdata,VV
size=100G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:15 sds 65:32 active ready running
SY1_log1_disk_001 (350002ac17cd4374a) dm-7 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:7 sdj 8:144 active ready running
MR2_log2_disk_001 (350002ac17ce7374a) dm-11 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:11 sdn 8:208 active ready running
MRC_disk_002 (350002ac2904a374a) dm-32 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:23 sdaa 65:160 active ready running
`- 1:0:0:23 sdcd 69:16 active ready running
SCR_ARCH_disk_001 (350002acb5a2c374a) dm-16 3PARdata,VV
size=30G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:17 sdu 65:64 active ready running
`- 1:0:0:17 sdca 68:224 active ready running
MR2_log1_disk_001 (350002ac17ce5374a) dm-12 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:12 sdo 8:224 active ready running
SY1_arch_disk_001 (350002ac17cca374a) dm-9 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:9 sdl 8:176 active ready running
SY1_disknew_001 (350002ac4c566374a) dm-4 3PARdata,VV
size=60G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:4 sdg 8:96 active ready running
SCR_LOG2_disk_001 (350002acb5a39374a) dm-17 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:18 sdv 65:80 active ready running
`- 1:0:0:18 sdq 65:0 active ready running
MR2_arch_disk_001 (350002ac17cd7374a) dm-15 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:14 sdr 65:16 active ready running
SCR_LOG1_disk_001 (350002acb5a38374a) dm-19 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:19 sdw 65:96 active ready running
`- 1:0:0:19 sdav 66:240 active ready running
MRC_data_disk_001 (350002ac15505374a) dm-2 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:2 sde 8:64 active ready running
sap_disk_001 (350002ac1fc92374a) dm-13 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:16 sdt 65:48 active ready running
SY1_disk_001 (350002ac17cbd374a) dm-10 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:10 sdm 8:192 active ready running
SY1_interf_disk_001 (350002ac0ce50374a) dm-33 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:24 sdab 65:176 active ready running
`- 1:0:0:24 sdce 69:32 active ready running
MRC_log2_disk_001 (350002ac1551a374a) dm-71 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:0 sdc 8:32 active ready running
SY1_data_disk_002 (350002ac23826374a) dm-31 3PARdata,VV
size=2.6T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:22 sdz 65:144 active ready running
`- 1:0:0:22 sdcc 69:0 active ready running
MRC_log1_disk_001 (350002ac15519374a) dm-72 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:1 sdd 8:48 active ready running
SCR_DATA_disk_001 (350002acb5a37374a) dm-23 3PARdata,VV
size=110G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:20 sdx 65:112 active ready running
SCR_disk_002 (350002ac19e73374a) dm-8 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:8 sdk 8:160 active ready running
swap_disk_001 (350002ac155d0374a) dm-5 3PARdata,VV
size=224G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:5 sdh 8:112 active ready running

Recognize new LUNs and run multipath -v3

root@linux:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | sh
root@linux:~ # multipath -v3

Paths restored

root@linux:~ # multipath -ll
SCR_disk_001 (350002acb5a22374a) dm-26 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:21 sdy 65:128 active ready running
`- 1:0:0:21 sdcb 68:240 active ready running
SY1_log2_disk_001 (350002ac17cd5374a) dm-6 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:6 sdi 8:128 active ready running
`- 1:0:0:6 sdai 66:32 active ready running
MRC_arch_disk_001 (350002ac15504374a) dm-3 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:3 sdf 8:80 active ready running
`- 1:0:0:3 sdaf 65:240 active ready running
MR2_data_disk_001 (350002ac17cd8374a) dm-14 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:13 sdp 8:240 active ready running
`- 1:0:0:13 sdap 66:144 active ready running
MR2_disk_001 (350002ac17cd6374a) dm-18 3PARdata,VV
size=100G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:15 sds 65:32 active ready running
`- 1:0:0:15 sdar 66:176 active ready running
SY1_log1_disk_001 (350002ac17cd4374a) dm-7 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:7 sdj 8:144 active ready running
`- 1:0:0:7 sdaj 66:48 active ready running
MR2_log2_disk_001 (350002ac17ce7374a) dm-11 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:11 sdn 8:208 active ready running
`- 1:0:0:11 sdan 66:112 active ready running
MRC_disk_002 (350002ac2904a374a) dm-32 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:23 sdaa 65:160 active ready running
`- 1:0:0:23 sdcd 69:16 active ready running
SCR_ARCH_disk_001 (350002acb5a2c374a) dm-16 3PARdata,VV
size=30G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:17 sdu 65:64 active ready running
`- 1:0:0:17 sdca 68:224 active ready running
MR2_log1_disk_001 (350002ac17ce5374a) dm-12 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:12 sdo 8:224 active ready running
`- 1:0:0:12 sdao 66:128 active ready running
SY1_arch_disk_001 (350002ac17cca374a) dm-9 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:9 sdl 8:176 active ready running
`- 1:0:0:9 sdal 66:80 active ready running
SY1_disknew_001 (350002ac4c566374a) dm-4 3PARdata,VV
size=60G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:4 sdg 8:96 active ready running
`- 1:0:0:4 sdag 66:0 active ready running
SCR_LOG2_disk_001 (350002acb5a39374a) dm-17 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:18 sdv 65:80 active ready running
`- 1:0:0:18 sdq 65:0 active ready running
MR2_arch_disk_001 (350002ac17cd7374a) dm-15 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:14 sdr 65:16 active ready running
`- 1:0:0:14 sdaq 66:160 active ready running
SCR_LOG1_disk_001 (350002acb5a38374a) dm-19 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:19 sdw 65:96 active ready running
`- 1:0:0:19 sdav 66:240 active ready running
MRC_data_disk_001 (350002ac15505374a) dm-2 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:2 sde 8:64 active ready running
`- 1:0:0:2 sdae 65:224 active ready running
sap_disk_001 (350002ac1fc92374a) dm-13 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:16 sdt 65:48 active ready running
`- 1:0:0:16 sdas 66:192 active ready running
SY1_disk_001 (350002ac17cbd374a) dm-10 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:10 sdm 8:192 active ready running
`- 1:0:0:10 sdam 66:96 active ready running
SY1_interf_disk_001 (350002ac0ce50374a) dm-33 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:24 sdab 65:176 active ready running
`- 1:0:0:24 sdce 69:32 active ready running
MRC_log2_disk_001 (350002ac1551a374a) dm-71 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:0 sdc 8:32 active ready running
`- 1:0:0:0 sdac 65:192 active ready running
SY1_data_disk_002 (350002ac23826374a) dm-31 3PARdata,VV
size=2.6T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:22 sdz 65:144 active ready running
`- 1:0:0:22 sdcc 69:0 active ready running
MRC_log1_disk_001 (350002ac15519374a) dm-72 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:1 sdd 8:48 active ready running
`- 1:0:0:1 sdad 65:208 active ready running
SCR_DATA_disk_001 (350002acb5a37374a) dm-23 3PARdata,VV
size=110G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:20 sdx 65:112 active ready running
`- 1:0:0:20 sdat 66:208 active ready running
SCR_disk_002 (350002ac19e73374a) dm-8 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:8 sdk 8:160 active ready running
`- 1:0:0:8 sdak 66:64 active ready running
swap_disk_001 (350002ac155d0374a) dm-5 3PARdata,VV
size=224G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:5 sdh 8:112 active ready running
`- 1:0:0:5 sdah 66:16 active ready running

Problem solved

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Fri Dec 15 11:39:25 2017 : INFO : UXMONmpmon is running now, pid=52191
mv: `/dev/null’ and `/dev/null’ are the same file
Fri Dec 15 11:39:26 2017 : INFO : UXMONmpmon end, pid=52191

One path missing in disk map on multipath device

Showing a particular case:

The disk mpath5 was only showing one path

root@linux:~ # multipath -ll mpath5
mpath5 (350002ac19430374a) dm-17 3PARdata,VV
[size=47G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 2:0:0:1 sdh 8:112 [active][ready]

The disk used by operating system is cciss/c0d0

root@linux:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/cciss/c0d0p3 vg00 lvm2 a– 269.47G 203.28G
/dev/mpath/350002ac19429374a vgapp lvm2 a– 100.00G 0
/dev/mpath/350002ac1942c374a vgapp lvm2 a– 20.00G 0
/dev/mpath/350002ac1942e374a vgapp lvm2 a– 75.00G 0
/dev/mpath/350002ac1942f374a vgapp lvm2 a– 158.00G 0
/dev/mpath/350002ac19430374a vgapp lvm2 a– 47.00G 996.00M
/dev/mpath/350002ac22869374a vgapp lvm2 a– 100.00G 0
/dev/mpath/350002ac2286a374a vgapp lvm2 a– 40.00G 0

Listing the SCSI devices. sda through sdn are used

root@linux:~ # lsscsi
[1:0:0:1] disk 3PARdata VV 3213 /dev/sda
[1:0:0:2] disk 3PARdata VV 3213 /dev/sdb
[1:0:0:3] disk 3PARdata VV 3213 /dev/sdc
[1:0:0:4] disk 3PARdata VV 3213 /dev/sdd
[1:0:0:5] disk 3PARdata VV 3213 /dev/sde
[1:0:0:6] disk 3PARdata VV 3213 /dev/sdf
[1:0:0:7] disk 3PARdata VV 3213 /dev/sdg
[1:0:0:254] enclosu 3PARdata SES 3213 –
[2:0:0:1] disk 3PARdata VV 3213 /dev/sdh
[2:0:0:2] disk 3PARdata VV 3213 /dev/sdi
[2:0:0:3] disk 3PARdata VV 3213 /dev/sdj
[2:0:0:4] disk 3PARdata VV 3213 /dev/sdk
[2:0:0:5] disk 3PARdata VV 3213 /dev/sdl
[2:0:0:6] disk 3PARdata VV 3213 /dev/sdm
[2:0:0:7] disk 3PARdata VV 3213 /dev/sdn
[2:0:0:254] enclosu 3PARdata SES 3213 –

Checking /etc/multipath.conf. sda was being blacklisted. Commented the line

root@linux:~ # grep -v ^# /etc/multipath.conf

blacklist {
devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”
devnode “^hd[a-z][[0-9]*]”
devnode “^hd[a-z]”
#devnode “^sda$”
}

defaults {
user_friendly_names yes
}

Running multipath -v3

root@linux:~ # multipath -v3

Checking disk mpath5

root@linux:~ # multipath -ll mpath5
mpath5 (350002ac19430374a) dm-17 3PARdata,VV
[size=47G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 1:0:0:1 sda 8:0 [active][ready]
\_ 2:0:0:1 sdh 8:112 [active][ready]

Adding new disks to Oracle ASM (Automatic Storage Management)

I have a two node Oracle RAC and I need to add disks to it

Storage team presented 3 new LUNs

rac1 & rac2
RAC1RAC2_PROJECT335066_Data1 (Tier1, RAID5, Size: 50GB)
World Wide LUN ID# 6001-4380-05de-d87b-0000-5000-10ef-0000
RAC1RAC2_PROJECT335066_Logs1 (Tier1, RAID1, Size: 20GB)
World Wide LUN ID# 6001-4380-05de-d87b-0000-5000-10f3-0000
RAC1RAC2_PROJECT335066_Quo (Tier1, RAID5, Size: 1GB)
World Wide LUN ID# 6001-4380-05de-d87b-0000-5000-10f7-0000

10ef
10f3
10f7

My server has a fibre-channel HBA card. Check HBA information. Install libsysfs and sysfsutils if you don’t have systool. Run yum install sysfsutils. I want to use an internal repository so I won’t use yum

root@rac1# systool -av -c fc_host
-bash: systool: command not found

root@rac1:~ # rpm -ivh http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/libsysfs-2.1.0-7.el6.x86_64.rpm http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/sysfsutils-2.1.0-7.el6.x86_64.rpm
Retrieving http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/libsysfs-2.1.0-7.el6.x86_64.rpm
Retrieving http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/sysfsutils-2.1.0-7.el6.x86_64.rpm
Preparing… ########################################### [100%]
1:libsysfs ########################################### [ 50%]
2:sysfsutils ########################################### [100%]

root@rac1:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘}
echo “- – -” > /sys/class/scsi_host/host0/scan
echo “- – -” > /sys/class/scsi_host/host1/scan

root@rac1:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | bash

Listing the disks under Oracle ASM

root@rac1:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001

Added the disk information to /etc/multipath.conf

multipath {
wwid 36001438005ded87b0000500010ef0000
alias asmdisk07
}
multipath {
wwid 36001438005ded87b0000500010f30000
alias asmdisk08
}
multipath {
wwid 36001438005ded87b0000500010f70000
alias votdisk04
}

Then run multipath -r to reload the new multipath aliases

Check if the aliases changed

root@rac1:~ # multipath -ll | grep 10ef
asmdisk07 (36001438005ded87b0000500010ef0000) dm-44 HP,HSV450
root@rac1:~ # multipath -ll | grep 10f3
asmdisk08 (36001438005ded87b0000500010f30000) dm-45 HP,HSV450
root@rac1:~ # multipath -ll | grep 10f7
votdisk04 (36001438005ded87b0000500010f70000) dm-46 HP,HSV450

DBA team asked to change owner and group for the disk devices

root@rac1:~ # chown oracle:dba /dev/mapper/asmdisk07
root@rac1:~ # chown oracle:dba /dev/mapper/asmdisk08
root@rac1:~ # chown oracle:dba /dev/mapper/votdisk04

Labeling disks

root@rac1:~ # /etc/init.d/oracleasm createdisk OCR_VOTE_006 votdisk04
Marking disk “OCR_VOTE_006” as an ASM disk: [ OK ]
root@rac1:~ # /etc/init.d/oracleasm createdisk ORADATA_002 asmdisk07
Marking disk “ORADATA_002” as an ASM disk: [ OK ]
root@rac1:~ # /etc/init.d/oracleasm createdisk ORADATA_003 asmdisk08
Marking disk “ORADATA_003” as an ASM disk: [ OK ]

Checking if the new disks

root@rac1:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001
ORADATA_002
ORADATA_003

In the other node, the disks are not updated automatically

root@rac2:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001

You need to perform the same steps and instead of running oracleasm disk, run oracleasm scandisks

root@rac2:~ # /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]

root@rac2:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001
ORADATA_002
ORADATA_003

Clustered Linux server showing device-mapper: multipath: Failing path in /var/log/messages

I have a disk presented to 4 servers.

Everyday we receive a notification saying that a specific multipathed disk lost all paths.

The disk is showing a SCSI reservation conflict – SCSI persistent reservations provide the capability to control the access of each node to shared storage devices

May 11 13:35:04 linux kernel: sd 0:0:0:38: reservation conflict
May 11 13:35:04 linux kernel: sd 0:0:0:38: [sdag] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 0:0:0:38: [sdag] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00

May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdag, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 66:0. <————————sdag
May 11 13:35:04 linux kernel: sd 1:0:0:38: reservation conflict
May 11 13:35:04 linux kernel: sd 1:0:0:38: [sdeh] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 1:0:0:38: [sdeh] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdeh, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 128:144. <———————-sdeh
May 11 13:35:04 linux multipathd: 66:0: mark as failed
May 11 13:35:04 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 3

May 11 13:35:04 linux kernel: sd 0:0:1:38: reservation conflict
May 11 13:35:04 linux kernel: sd 0:0:1:38: [sdcc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 0:0:1:38: [sdcc] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdcc, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 69:0.<—————————sdcc
May 11 13:35:04 linux kernel: sd 1:0:1:38: reservation conflict
May 11 13:35:04 linux kernel: sd 1:0:1:38: [sdgg] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 1:0:1:38: [sdgg] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdgg, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 131:192.<————————-sdgg
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev dm-209, sector 5200

May 11 13:35:05 linux multipathd: 128:144: mark as failed
May 11 13:35:05 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 2
May 11 13:35:05 linux multipathd: 69:0: mark as failed
May 11 13:35:05 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 1
May 11 13:35:05 linux multipathd: 131:192: mark as failed

PP0_oraarch_disk_001 (350002ad05071374b) dm-209 3PARdata,VV
size=300G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 0:0:0:38 sdag 66:0 active undef running
|- 1:0:0:38 sdeh 128:144 active undef running
|- 0:0:1:38 sdcc 69:0 active undef running
`- 1:0:1:38 sdgg 131:192 active undef running

Since this is opening support tickets, I removed monitoring for this disk

LVM – Disk removed and appears message read failed after 0 of 4096 at 4096 or similar

In this server when I ran pvs it was showing a lot of input/output errors

root@linux:~ # pvs
/dev/sdq: read failed after 0 of 4096 at 0: Input/output error
/dev/sdq1: read failed after 0 of 2048 at 0: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 80530571264: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 80530628608: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 0: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 4096: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 80525328384: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 80525447168: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 0: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 4096: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 2048 at 0: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 21474770944: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 21474828288: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 0: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 4096: Input/output error
/dev/sde: read failed after 0 of 4096 at 0: Input/output error
/dev/sde1: read failed after 0 of 2048 at 0: Input/output error
/dev/sdw: read failed after 0 of 4096 at 0: Input/output error
/dev/sdw1: read failed after 0 of 2048 at 0: Input/output error
/dev/sdk: read failed after 0 of 4096 at 0: Input/output error

A disk was removed and it was not removed cleanly

root@linux:~ # multipath -ll disk2
sde: checker msg is “tur checker reports path is down”
sdk: checker msg is “tur checker reports path is down”
sdq: checker msg is “tur checker reports path is down”
sdw: checker msg is “tur checker reports path is down”
disk2 (360000000000000000000000000000000) dm-25 HP,P2000 G3 FC
[size=75G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:5 sde 8:64 [failed][faulty]
\_ 1:0:1:5 sdk 8:160 [failed][faulty]
\_ 2:0:0:5 sdq 65:0 [failed][faulty]
\_ 2:0:1:5 sdw 65:96 [failed][faulty]

Removed the block devices

root@linux:~ # echo 1 > /sys/block/sde/device/delete
root@linux:~ # echo 1 > /sys/block/sdk/device/delete
root@linux:~ # echo 1 > /sys/block/sdq/device/delete
root@linux:~ # echo 1 > /sys/block/sdw/device/delete

I was able to remove the device that device-mapper used but I was not able to remove the device with the partition

root@linux:~ # dmsetup remove disk2
root@linux:~ # dmsetup remove disk2p1
device-mapper: remove ioctl failed: Device or resource busy
Command failed

I had a filesystem using the device /dev/devvg/u01lv so I had to remove it before I was able to remove the device with the partition

root@linux:~ # dmsetup remove devvg-u01lv

root@linux:~ # dmsetup remove disk2p1

UXMON: mpathb – Only one path detected, no path redundancy

Also see:
UXMON: volumegroup – Only one path detected, no path redundancy
UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2015-10-14 12:39:19
Message : UXMON: mpathb – Only one path detected, no path redundancy
Msg Group : OS
Application : mpmon
Object : mp
Event Type :
not_found

Instance Name :
not_found

Instruction : The multipathd -k”show map $device topology” command shows more details

Please check /var/opt/OV/log/OpC/mp_mon.log for more details

Checking the log file it complains about the mpathb

root@linux:~ # cat /var/opt/OV/log/OpC/mp_mon.log
Wed Oct 14 13:39:13 2015 : INFO : UXMONmpmon is running now, pid=21954
Wed Oct 14 13:39:13 2015 : Major: mpathb – Only one path detected, no path redundancy
Wed Oct 14 13:39:13 2015 : INFO : UXMONmpmon end, pid=21954
Wed Oct 14 13:56:12 2015 : INFO : UXMONmpmon is running now, pid=29130
Wed Oct 14 13:56:12 2015 : Major: mpathb – Only one path detected, no path redundancy
Wed Oct 14 13:56:12 2015 : INFO : UXMONmpmon end, pid=29130
Wed Oct 14 14:13:13 2015 : INFO : UXMONmpmon is running now, pid=36813
Wed Oct 14 14:13:13 2015 : Major: mpathb – Only one path detected, no path redundancy
Wed Oct 14 14:13:13 2015 : INFO : UXMONmpmon end, pid=36813
Wed Oct 14 14:30:13 2015 : INFO : UXMONmpmon is running now, pid=44029
Wed Oct 14 14:30:13 2015 : Major: mpathb – Only one path detected, no path redundancy
Wed Oct 14 14:30:13 2015 : INFO : UXMONmpmon end, pid=44029
Wed Oct 14 14:47:12 2015 : INFO : UXMONmpmon is running now, pid=51897
Wed Oct 14 14:47:13 2015 : INFO : UXMONmpmon end, pid=51897
Wed Oct 14 15:04:12 2015 : INFO : UXMONmpmon is running now, pid=58833
Wed Oct 14 15:04:12 2015 : INFO : UXMONmpmon end, pid=58833

In this server it is a local disk so it was added to the multipath blacklist

root@linux:~ # vi /etc/multipath.conf
blacklist {
devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”
devnode “^hd[a-z]”
devnode “^sd[ab]$”
devnode “^cciss!c[0-9]d[0-9]*”
}

If you are in a VMware host, you can safely disable this module.

root@linux:~ # cp /var/opt/OV/bin/instrumentation/mp_mon.cfg /var/opt/OV/conf/OpC/

In the configuration file /var/opt/OV/conf/OpC/mp_mon.cfg set disable to yes

root@linux:~ # vi /var/opt/OV/conf/OpC/mp_mon.cfg
disable = yes

multipath: /sbin/scsi_id exitted with 1 – cannot get the the wwid for cciss!c0d0

Whenever you run multipath and shows the message cannot get the the wwid for cciss!c0d0

root@linux:~ # multipath -ll oradisk004
/sbin/scsi_id exitted with 1
cannot get the the wwid for cciss!c0d0
oradisk004 (360060e800573b800000073b8000012d2) dm-12 HP,OPEN-V
[size=50G][features=1 queue_if_no_path][hwhandler=1 hp-sw][rw]
\_ round-robin 0 [prio=8][active]
\_ 1:0:0:3 sdd 8:48 [active][ready]
\_ 2:0:0:3 sdh 8:112 [active][ready]

Edit file /etc/multipath.conf and verify if you are blacklisting the cciss drive

blacklist {
devnode “^cciss!c[0-9]d[0-9]*”
}