Tag: systool

UXMON: mpatho – Only one FC Switch detected, no SAN FC Switch redundancy

After receiving the information that the error was caused by storage node reboot, checking if HPOM still show the error

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Thu May 14 10:34:20 2020 : INFO : UXMONmpmon is running now, pid=31809
Thu May 14 10:34:20 2020 : Minor: mpatho – Only one FC Switch detected, no SAN FC Switch redundancy
mv: `/dev/null’ and `/dev/null’ are the same file
Thu May 14 10:34:20 2020 : INFO : UXMONmpmon end, pid=31809

Checking multipath for mpatho

root@linux:~ # multipath -ll mpatho
mpatho (350002ac666af374b) dm-74 3PARdata,VV
size=150G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 2:0:0:5 sday 67:32 active ready running
`- 4:0:0:5 sdax 67:16 active ready running

Rescaning server

root@linux:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | bash

Checking multipath for mpatho

root@linux:~ # multipath -ll mpatho
mpatho (350002ac666af374b) dm-74 3PARdata,VV
size=150G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 2:0:0:5 sday 67:32 active ready running
|- 4:0:0:5 sdax 67:16 active ready running
|- 1:0:0:5 sdaz 67:48 active ready running
`- 3:0:0:5 sdba 67:64 active ready running

Checking if HPOM still show the error

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Thu May 14 10:36:26 2020 : INFO : UXMONmpmon is running now, pid=37696
mv: `/dev/null’ and `/dev/null’ are the same file
Thu May 14 10:36:26 2020 : INFO : UXMONmpmon end, pid=37696

UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy

Also see:
UXMON: mpathb – Only one path detected, no path redundancy
UXMON: volumegroup – Only one path detected, no path redundancy

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2017-12-11 03:28:31
Message : UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy
Msg Group : OS
Application : mpmon
Object : mp
Event Type :
not_found

Instance Name :
not_found

Instruction : The multipathd -k”show map $device topology” command shows more details

Please check /var/opt/OV/log/OpC/mp_mon.log for more details
EventDataSource :

When running a LVM command it is showing that several PVs are duplicate.

root@linux:~ # pvs
Found duplicate PV s3zEE42awIhydJ05hfUzJulPsN8WS266: using /dev/mapper/SY1_disknew_001 not /dev/sdbm
Using duplicate PV /dev/mapper/SY1_disknew_001 from subsystem DM, replacing /dev/sdbm
Found duplicate PV zTZlnwYgW69xzUcTi0riu5euTPiWWnRs: using /dev/mapper/swap_disk_001p1 not /dev/sdbo1
Using duplicate PV /dev/mapper/swap_disk_001p1 from subsystem DM, ignoring /dev/sdbo1
Found duplicate PV hnuWyziFeXRhJ71YpLJX1cLAjOvgtl0q: using /dev/mapper/SCR_DATA_disk_001p1 not /dev/sdbn1
Using duplicate PV /dev/mapper/SCR_DATA_disk_001p1 from subsystem DM, replacing /dev/sdbn1
Found duplicate PV EeqL0LDdMohfnagni16NlRUig3eugbap: using /dev/mapper/SY1_log2_disk_001p1 not /dev/sdbp1
Using duplicate PV /dev/mapper/SY1_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbp1
Found duplicate PV sMAYQABFedTStD589d5ZcDu1ZNtXfXyh: using /dev/mapper/SY1_log1_disk_001p1 not /dev/sdbq1
Using duplicate PV /dev/mapper/SY1_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbq1
Found duplicate PV qFBDDPa7FtN7F97tpUDeKv0cBO94WBr3: using /dev/mapper/SCR_disk_002 not /dev/sdbr
Using duplicate PV /dev/mapper/SCR_disk_002 from subsystem DM, ignoring /dev/sdbr
Found duplicate PV RZvwCnQn48G2A9IifNOLZT9l9ZYdE7yu: using /dev/mapper/SY1_arch_disk_001p1 not /dev/sdbs1
Using duplicate PV /dev/mapper/SY1_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbs1
Found duplicate PV qWPh0v3E1ADowxUuuC8kP8BFJ1bwBsmL: using /dev/mapper/SY1_disk_001p1 not /dev/sdbt1
Using duplicate PV /dev/mapper/SY1_disk_001p1 from subsystem DM, ignoring /dev/sdbt1
Found duplicate PV hi5F4g9FPojaD9J7KH5vHCzRKgd4jQfJ: using /dev/mapper/MR2_log2_disk_001p1 not /dev/sdbu1
Using duplicate PV /dev/mapper/MR2_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbu1
Found duplicate PV KA1AkbCOgfHFGuKazmHAacwqBNPmEyV8: using /dev/mapper/MR2_log1_disk_001p1 not /dev/sdbv1
Using duplicate PV /dev/mapper/MR2_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbv1
Found duplicate PV dy0YaJoSrlZdTM9isfY1QGS6kWBTzs6i: using /dev/mapper/MR2_data_disk_001p1 not /dev/sdbw1
Using duplicate PV /dev/mapper/MR2_data_disk_001p1 from subsystem DM, ignoring /dev/sdbw1
Found duplicate PV khSEnsJ9CC4epw0uOAsaFwlbIKS32Qyj: using /dev/mapper/MR2_arch_disk_001p1 not /dev/sdbx1
Using duplicate PV /dev/mapper/MR2_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbx1
Found duplicate PV PmTTDGWvEvDM8AMUHvqcoRJeA9g4r78D: using /dev/mapper/MRC_log2_disk_001p1 not /dev/sdbi1
Using duplicate PV /dev/mapper/MRC_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbi1
Found duplicate PV ffv1TgDxRIEIyhRoEh46fMGeFuJ1tVtv: using /dev/mapper/MR2_disk_001p1 not /dev/sdby1
Using duplicate PV /dev/mapper/MR2_disk_001p1 from subsystem DM, ignoring /dev/sdby1
Found duplicate PV tCeMdOB1dFuJomhgB061M7MeZNEwptk3: using /dev/mapper/MRC_log1_disk_001p1 not /dev/sdbj1
Using duplicate PV /dev/mapper/MRC_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbj1
Found duplicate PV inuuS3f39VjSuh6q7r5pEZSjzRgRhi5e: using /dev/mapper/sap_disk_001p1 not /dev/sdbz1
Using duplicate PV /dev/mapper/sap_disk_001p1 from subsystem DM, ignoring /dev/sdbz1
Found duplicate PV Ea4CVYCMjYc3eE129uYlUlZKdzhmYNpQ: using /dev/mapper/MRC_data_disk_001p1 not /dev/sdbk1
Using duplicate PV /dev/mapper/MRC_data_disk_001p1 from subsystem DM, ignoring /dev/sdbk1
Found duplicate PV Lx5viH3geNo0zSO07sHcbNlJ9nVmcGPA: using /dev/mapper/MRC_arch_disk_001p1 not /dev/sdbl1
Using duplicate PV /dev/mapper/MRC_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbl1
PV VG Fmt Attr PSize PFree
/dev/mapper/MR2_arch_disk_001p1 vgMR2oraarch lvm2 a–u 19.98g 0
/dev/mapper/MR2_data_disk_001p1 vgMR2data lvm2 a–u 299.98g 99.98g
/dev/mapper/MR2_disk_001p1 vgMR2 lvm2 a–u 99.98g 20.87g
/dev/mapper/MR2_log1_disk_001p1 vgMR2log1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MR2_log2_disk_001p1 vgMR2log2 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_arch_disk_001p1 vgMRCoraarch lvm2 a–u 19.98g 0
/dev/mapper/MRC_data_disk_001p1 vgMRCdata lvm2 a–u 299.98g 39.98g
/dev/mapper/MRC_disk_002 vgMRC lvm2 a–u 149.98g 31.88g
/dev/mapper/MRC_log1_disk_001p1 vgMRClog1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_log2_disk_001p1 vgMRClog2 lvm2 a–u 9.98g 7.98g
/dev/mapper/SCR_ARCH_disk_001p1 vgSCRarch lvm2 a–u 29.96g 4.57g
/dev/mapper/SCR_DATA_disk_001p1 vgSCRdata lvm2 a–u 109.96g 8.40g
/dev/mapper/SCR_LOG1_disk_001p1 vgSCRlog1 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_LOG2_disk_001p1 vgSCRlog2 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_disk_001p1 vgSCR lvm2 a–u 49.96g 3.96g
/dev/mapper/SCR_disk_002 vgSCR lvm2 a–u 49.98g 34.98g
/dev/mapper/SY1_arch_disk_001p1 vgSY1oraarch lvm2 a–u 149.98g 0
/dev/mapper/SY1_data_disk_002 vgSY1data lvm2 a–u 2.64t 0
/dev/mapper/SY1_disk_001p1 vgSY1 lvm2 a–u 49.98g 10.52g
/dev/mapper/SY1_disknew_001 vgSY1 lvm2 a–u 59.98g 44.98g
/dev/mapper/SY1_interf_disk_001 vgSY1interface lvm2 a–u 49.98g 1008.00m
/dev/mapper/SY1_log1_disk_001p1 vgSY1log1 lvm2 a–u 19.98g 9.98g
/dev/mapper/SY1_log2_disk_001p1 vgSY1log2 lvm2 a–u 19.98g 9.98g
/dev/mapper/sap_disk_001p1 vgSAPlocal lvm2 a–u 49.98g 4.79g
/dev/mapper/swap_disk_001p1 vgswap lvm2 a–u 223.98g 0
/dev/sda2 vgroot lvm2 a–u 278.86g 198.27g

Removing disk paths that are duplicate

echo 1 > /sys/block/sdbm/device/delete
echo 1 > /sys/block/sdbo/device/delete
echo 1 > /sys/block/sdbn/device/delete
echo 1 > /sys/block/sdbp/device/delete
echo 1 > /sys/block/sdbq/device/delete
echo 1 > /sys/block/sdbr/device/delete
echo 1 > /sys/block/sdbs/device/delete
echo 1 > /sys/block/sdbt/device/delete
echo 1 > /sys/block/sdbu/device/delete
echo 1 > /sys/block/sdbv/device/delete
echo 1 > /sys/block/sdbw/device/delete
echo 1 > /sys/block/sdbx/device/delete
echo 1 > /sys/block/sdbi/device/delete
echo 1 > /sys/block/sdby/device/delete
echo 1 > /sys/block/sdbj/device/delete
echo 1 > /sys/block/sdbz/device/delete

Checking status

root@linux:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/MR2_arch_disk_001p1 vgMR2oraarch lvm2 a–u 19.98g 0
/dev/mapper/MR2_data_disk_001p1 vgMR2data lvm2 a–u 299.98g 99.98g
/dev/mapper/MR2_disk_001p1 vgMR2 lvm2 a–u 99.98g 20.87g
/dev/mapper/MR2_log1_disk_001p1 vgMR2log1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MR2_log2_disk_001p1 vgMR2log2 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_arch_disk_001p1 vgMRCoraarch lvm2 a–u 19.98g 0
/dev/mapper/MRC_data_disk_001p1 vgMRCdata lvm2 a–u 299.98g 39.98g
/dev/mapper/MRC_disk_002 vgMRC lvm2 a–u 149.98g 31.88g
/dev/mapper/MRC_log1_disk_001p1 vgMRClog1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_log2_disk_001p1 vgMRClog2 lvm2 a–u 9.98g 7.98g
/dev/mapper/SCR_ARCH_disk_001p1 vgSCRarch lvm2 a–u 29.96g 4.57g
/dev/mapper/SCR_DATA_disk_001p1 vgSCRdata lvm2 a–u 109.96g 8.40g
/dev/mapper/SCR_LOG1_disk_001p1 vgSCRlog1 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_LOG2_disk_001p1 vgSCRlog2 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_disk_001p1 vgSCR lvm2 a–u 49.96g 3.96g
/dev/mapper/SCR_disk_002 vgSCR lvm2 a–u 49.98g 34.98g
/dev/mapper/SY1_arch_disk_001p1 vgSY1oraarch lvm2 a–u 149.98g 0
/dev/mapper/SY1_data_disk_002 vgSY1data lvm2 a–u 2.64t 0
/dev/mapper/SY1_disk_001p1 vgSY1 lvm2 a–u 49.98g 10.52g
/dev/mapper/SY1_disknew_001 vgSY1 lvm2 a–u 59.98g 44.98g
/dev/mapper/SY1_interf_disk_001 vgSY1interface lvm2 a–u 49.98g 1008.00m
/dev/mapper/SY1_log1_disk_001p1 vgSY1log1 lvm2 a–u 19.98g 9.98g
/dev/mapper/SY1_log2_disk_001p1 vgSY1log2 lvm2 a–u 19.98g 9.98g
/dev/mapper/sap_disk_001p1 vgSAPlocal lvm2 a–u 49.98g 4.79g
/dev/mapper/swap_disk_001p1 vgswap lvm2 a–u 223.98g 0
/dev/sda2 vgroot lvm2 a–u 278.86g 198.27g

Checking multipath devices. Some devices have only one path

root@linux:~ # multipath -ll
SCR_disk_001 (350002acb5a22374a) dm-26 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:21 sdy 65:128 active ready running
`- 1:0:0:21 sdcb 68:240 active ready running
SY1_log2_disk_001 (350002ac17cd5374a) dm-6 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:6 sdi 8:128 active ready running
MRC_arch_disk_001 (350002ac15504374a) dm-3 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:3 sdf 8:80 active ready running
MR2_data_disk_001 (350002ac17cd8374a) dm-14 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:13 sdp 8:240 active ready running
MR2_disk_001 (350002ac17cd6374a) dm-18 3PARdata,VV
size=100G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:15 sds 65:32 active ready running
SY1_log1_disk_001 (350002ac17cd4374a) dm-7 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:7 sdj 8:144 active ready running
MR2_log2_disk_001 (350002ac17ce7374a) dm-11 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:11 sdn 8:208 active ready running
MRC_disk_002 (350002ac2904a374a) dm-32 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:23 sdaa 65:160 active ready running
`- 1:0:0:23 sdcd 69:16 active ready running
SCR_ARCH_disk_001 (350002acb5a2c374a) dm-16 3PARdata,VV
size=30G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:17 sdu 65:64 active ready running
`- 1:0:0:17 sdca 68:224 active ready running
MR2_log1_disk_001 (350002ac17ce5374a) dm-12 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:12 sdo 8:224 active ready running
SY1_arch_disk_001 (350002ac17cca374a) dm-9 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:9 sdl 8:176 active ready running
SY1_disknew_001 (350002ac4c566374a) dm-4 3PARdata,VV
size=60G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:4 sdg 8:96 active ready running
SCR_LOG2_disk_001 (350002acb5a39374a) dm-17 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:18 sdv 65:80 active ready running
`- 1:0:0:18 sdq 65:0 active ready running
MR2_arch_disk_001 (350002ac17cd7374a) dm-15 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:14 sdr 65:16 active ready running
SCR_LOG1_disk_001 (350002acb5a38374a) dm-19 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:19 sdw 65:96 active ready running
`- 1:0:0:19 sdav 66:240 active ready running
MRC_data_disk_001 (350002ac15505374a) dm-2 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:2 sde 8:64 active ready running
sap_disk_001 (350002ac1fc92374a) dm-13 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:16 sdt 65:48 active ready running
SY1_disk_001 (350002ac17cbd374a) dm-10 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:10 sdm 8:192 active ready running
SY1_interf_disk_001 (350002ac0ce50374a) dm-33 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:24 sdab 65:176 active ready running
`- 1:0:0:24 sdce 69:32 active ready running
MRC_log2_disk_001 (350002ac1551a374a) dm-71 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:0 sdc 8:32 active ready running
SY1_data_disk_002 (350002ac23826374a) dm-31 3PARdata,VV
size=2.6T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:22 sdz 65:144 active ready running
`- 1:0:0:22 sdcc 69:0 active ready running
MRC_log1_disk_001 (350002ac15519374a) dm-72 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:1 sdd 8:48 active ready running
SCR_DATA_disk_001 (350002acb5a37374a) dm-23 3PARdata,VV
size=110G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:20 sdx 65:112 active ready running
SCR_disk_002 (350002ac19e73374a) dm-8 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:8 sdk 8:160 active ready running
swap_disk_001 (350002ac155d0374a) dm-5 3PARdata,VV
size=224G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:5 sdh 8:112 active ready running

Recognize new LUNs and run multipath -v3

root@linux:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | sh
root@linux:~ # multipath -v3

Paths restored

root@linux:~ # multipath -ll
SCR_disk_001 (350002acb5a22374a) dm-26 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:21 sdy 65:128 active ready running
`- 1:0:0:21 sdcb 68:240 active ready running
SY1_log2_disk_001 (350002ac17cd5374a) dm-6 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:6 sdi 8:128 active ready running
`- 1:0:0:6 sdai 66:32 active ready running
MRC_arch_disk_001 (350002ac15504374a) dm-3 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:3 sdf 8:80 active ready running
`- 1:0:0:3 sdaf 65:240 active ready running
MR2_data_disk_001 (350002ac17cd8374a) dm-14 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:13 sdp 8:240 active ready running
`- 1:0:0:13 sdap 66:144 active ready running
MR2_disk_001 (350002ac17cd6374a) dm-18 3PARdata,VV
size=100G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:15 sds 65:32 active ready running
`- 1:0:0:15 sdar 66:176 active ready running
SY1_log1_disk_001 (350002ac17cd4374a) dm-7 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:7 sdj 8:144 active ready running
`- 1:0:0:7 sdaj 66:48 active ready running
MR2_log2_disk_001 (350002ac17ce7374a) dm-11 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:11 sdn 8:208 active ready running
`- 1:0:0:11 sdan 66:112 active ready running
MRC_disk_002 (350002ac2904a374a) dm-32 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:23 sdaa 65:160 active ready running
`- 1:0:0:23 sdcd 69:16 active ready running
SCR_ARCH_disk_001 (350002acb5a2c374a) dm-16 3PARdata,VV
size=30G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:17 sdu 65:64 active ready running
`- 1:0:0:17 sdca 68:224 active ready running
MR2_log1_disk_001 (350002ac17ce5374a) dm-12 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:12 sdo 8:224 active ready running
`- 1:0:0:12 sdao 66:128 active ready running
SY1_arch_disk_001 (350002ac17cca374a) dm-9 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:9 sdl 8:176 active ready running
`- 1:0:0:9 sdal 66:80 active ready running
SY1_disknew_001 (350002ac4c566374a) dm-4 3PARdata,VV
size=60G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:4 sdg 8:96 active ready running
`- 1:0:0:4 sdag 66:0 active ready running
SCR_LOG2_disk_001 (350002acb5a39374a) dm-17 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:18 sdv 65:80 active ready running
`- 1:0:0:18 sdq 65:0 active ready running
MR2_arch_disk_001 (350002ac17cd7374a) dm-15 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:14 sdr 65:16 active ready running
`- 1:0:0:14 sdaq 66:160 active ready running
SCR_LOG1_disk_001 (350002acb5a38374a) dm-19 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:19 sdw 65:96 active ready running
`- 1:0:0:19 sdav 66:240 active ready running
MRC_data_disk_001 (350002ac15505374a) dm-2 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:2 sde 8:64 active ready running
`- 1:0:0:2 sdae 65:224 active ready running
sap_disk_001 (350002ac1fc92374a) dm-13 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:16 sdt 65:48 active ready running
`- 1:0:0:16 sdas 66:192 active ready running
SY1_disk_001 (350002ac17cbd374a) dm-10 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:10 sdm 8:192 active ready running
`- 1:0:0:10 sdam 66:96 active ready running
SY1_interf_disk_001 (350002ac0ce50374a) dm-33 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:24 sdab 65:176 active ready running
`- 1:0:0:24 sdce 69:32 active ready running
MRC_log2_disk_001 (350002ac1551a374a) dm-71 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:0 sdc 8:32 active ready running
`- 1:0:0:0 sdac 65:192 active ready running
SY1_data_disk_002 (350002ac23826374a) dm-31 3PARdata,VV
size=2.6T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:22 sdz 65:144 active ready running
`- 1:0:0:22 sdcc 69:0 active ready running
MRC_log1_disk_001 (350002ac15519374a) dm-72 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:1 sdd 8:48 active ready running
`- 1:0:0:1 sdad 65:208 active ready running
SCR_DATA_disk_001 (350002acb5a37374a) dm-23 3PARdata,VV
size=110G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:20 sdx 65:112 active ready running
`- 1:0:0:20 sdat 66:208 active ready running
SCR_disk_002 (350002ac19e73374a) dm-8 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:8 sdk 8:160 active ready running
`- 1:0:0:8 sdak 66:64 active ready running
swap_disk_001 (350002ac155d0374a) dm-5 3PARdata,VV
size=224G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:5 sdh 8:112 active ready running
`- 1:0:0:5 sdah 66:16 active ready running

Problem solved

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Fri Dec 15 11:39:25 2017 : INFO : UXMONmpmon is running now, pid=52191
mv: `/dev/null’ and `/dev/null’ are the same file
Fri Dec 15 11:39:26 2017 : INFO : UXMONmpmon end, pid=52191

Linux Fibre HBA in status Linkdown

root@linux:~ # systool -c fc_host -v
Class = “fc_host”

Class Device = “host1”
Class Device path = “/sys/class/fc_host/host1”
fabric_name = “0x100000051e36219e”
issue_lip =
node_name = “0x50060b0000c29e39”
port_id = “0x1ebe01”
port_name = “0x50060b0000c29e38”
port_state = “Online”
port_type = “NPort (fabric via point-to-point)”
speed = “4 Gbit”
supported_classes = “Class 3”
supported_speeds = “1 Gbit, 2 Gbit, 4 Gbit”
symbolic_name = “QMH2462 FW:v5.06.03 DVR:v8.03.07.15.05.09-k”
system_hostname = “”
tgtid_bind_type = “wwpn (World Wide Port Name)”
uevent =

Device = “host1”
Device path = “/sys/devices/pci0000:00/0000:00:09.0/0000:09:00.0/host1”
fw_dump =
nvram = “ISP ”
optrom_ctl =
optrom =
reset =
uevent =
vpd = “▒8”

Class Device = “host2”
Class Device path = “/sys/class/fc_host/host2”
fabric_name = “0x50060b0000c29e3b”
issue_lip =
node_name = “0x50060b0000c29e3b”
port_id = “0x000000”
port_name = “0x50060b0000c29e3a”
port_state = “Linkdown”
port_type = “Unknown”
speed = “unknown”
supported_classes = “Class 3”
supported_speeds = “1 Gbit, 2 Gbit, 4 Gbit”
symbolic_name = “QMH2462 FW:v5.06.03 DVR:v8.03.07.15.05.09-k”
system_hostname = “”
tgtid_bind_type = “wwpn (World Wide Port Name)”
uevent =

Device = “host2”
Device path = “/sys/devices/pci0000:00/0000:00:09.0/0000:09:00.1/host2”
fw_dump =
nvram = “ISP ”
optrom_ctl =
optrom =
reset =
uevent =
vpd = “▒8”

Perform diagnostic steps below to isolate initialization issue. Engaged hardware vendor to check the card

Adding new disks to Oracle ASM (Automatic Storage Management)

I have a two node Oracle RAC and I need to add disks to it

Storage team presented 3 new LUNs

rac1 & rac2
RAC1RAC2_PROJECT335066_Data1 (Tier1, RAID5, Size: 50GB)
World Wide LUN ID# 6001-4380-05de-d87b-0000-5000-10ef-0000
RAC1RAC2_PROJECT335066_Logs1 (Tier1, RAID1, Size: 20GB)
World Wide LUN ID# 6001-4380-05de-d87b-0000-5000-10f3-0000
RAC1RAC2_PROJECT335066_Quo (Tier1, RAID5, Size: 1GB)
World Wide LUN ID# 6001-4380-05de-d87b-0000-5000-10f7-0000

10ef
10f3
10f7

My server has a fibre-channel HBA card. Check HBA information. Install libsysfs and sysfsutils if you don’t have systool. Run yum install sysfsutils. I want to use an internal repository so I won’t use yum

root@rac1# systool -av -c fc_host
-bash: systool: command not found

root@rac1:~ # rpm -ivh http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/libsysfs-2.1.0-7.el6.x86_64.rpm http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/sysfsutils-2.1.0-7.el6.x86_64.rpm
Retrieving http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/libsysfs-2.1.0-7.el6.x86_64.rpm
Retrieving http://172.22.19.185/rhel/redhat/rhel-x86_64-server-6/getPackage/sysfsutils-2.1.0-7.el6.x86_64.rpm
Preparing… ########################################### [100%]
1:libsysfs ########################################### [ 50%]
2:sysfsutils ########################################### [100%]

root@rac1:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘}
echo “- – -” > /sys/class/scsi_host/host0/scan
echo “- – -” > /sys/class/scsi_host/host1/scan

root@rac1:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | bash

Listing the disks under Oracle ASM

root@rac1:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001

Added the disk information to /etc/multipath.conf

multipath {
wwid 36001438005ded87b0000500010ef0000
alias asmdisk07
}
multipath {
wwid 36001438005ded87b0000500010f30000
alias asmdisk08
}
multipath {
wwid 36001438005ded87b0000500010f70000
alias votdisk04
}

Then run multipath -r to reload the new multipath aliases

Check if the aliases changed

root@rac1:~ # multipath -ll | grep 10ef
asmdisk07 (36001438005ded87b0000500010ef0000) dm-44 HP,HSV450
root@rac1:~ # multipath -ll | grep 10f3
asmdisk08 (36001438005ded87b0000500010f30000) dm-45 HP,HSV450
root@rac1:~ # multipath -ll | grep 10f7
votdisk04 (36001438005ded87b0000500010f70000) dm-46 HP,HSV450

DBA team asked to change owner and group for the disk devices

root@rac1:~ # chown oracle:dba /dev/mapper/asmdisk07
root@rac1:~ # chown oracle:dba /dev/mapper/asmdisk08
root@rac1:~ # chown oracle:dba /dev/mapper/votdisk04

Labeling disks

root@rac1:~ # /etc/init.d/oracleasm createdisk OCR_VOTE_006 votdisk04
Marking disk “OCR_VOTE_006” as an ASM disk: [ OK ]
root@rac1:~ # /etc/init.d/oracleasm createdisk ORADATA_002 asmdisk07
Marking disk “ORADATA_002” as an ASM disk: [ OK ]
root@rac1:~ # /etc/init.d/oracleasm createdisk ORADATA_003 asmdisk08
Marking disk “ORADATA_003” as an ASM disk: [ OK ]

Checking if the new disks

root@rac1:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001
ORADATA_002
ORADATA_003

In the other node, the disks are not updated automatically

root@rac2:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001

You need to perform the same steps and instead of running oracleasm disk, run oracleasm scandisks

root@rac2:~ # /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]

root@rac2:~ # /etc/init.d/oracleasm listdisks
OCR_VOTE_001
OCR_VOTE_002
OCR_VOTE_003
OCR_VOTE_004
OCR_VOTE_005
ORAARCH_001
ORADATA_001
ORADATA_002
ORADATA_003

systool – Query information about Fibre Channel HBA card on a Linux server

To query information about Fibre Channel HBA cards, use systool

root@linux:~ # which systool
/usr/bin/systool

Install the package sysfsutils using yum install systool if you have a Red Hat based distribution

root@linux:~ # rpm -qf /usr/bin/systool
sysfsutils-2.1.0-7.el6.x86_64

root@linux:~ # rpm -qi sysfsutils
Name : sysfsutils Relocations: (not relocatable)
Version : 2.1.0 Vendor: Red Hat, Inc.
Release : 7.el6 Build Date: Thu 20 Sep 2012 10:11:43 AM BRT
Install Date: Fri 23 May 2014 12:17:31 PM BRT Build Host: x86-012.build.bos.redhat.com
Group : Development/Tools Source RPM: sysfsutils-2.1.0-7.el6.src.rpm
Size : 117100 License: GPLv2
Signature : RSA/8, Fri 09 Nov 2012 09:27:58 AM BRST, Key ID 199e2f91fd431d51
Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla&gt;
URL : http://sourceforge.net/projects/linux-diag/
Summary : Utilities for interfacing with sysfs
Description :
This package’s purpose is to provide a set of utilities for interfacing
with sysfs.

root@linux:~ # cat /etc/*release
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Red Hat Enterprise Linux Server release 6.5 (Santiago)
Red Hat Enterprise Linux Server release 6.5 (Santiago)

Updating HP BLc Emulex LPe1105-HP FC Mezz Option Kit – Flash Engine exit code: 2 – All updates failed. – Flash complete [7]

I was receiving error Flash Engine exit code: 2

root@suse11:/usr/lib/x86_64-linux-gnu/hp-firmware-fc-emulex-2015.02.01-1.1 # ./.hpsetup
—————————————————
Loading references…done.
Performing discovery…
Preparing the Emulex Kit…done
Performing flash — DO NOT INTERRUPT!
Executing flash engine: ./elxflashStandalone*/lx/elxflash.sh /fc /update 1>> /var/cpq/Component.log 2>> /var/cpq/Component.log 3>> /var/cpq/Component.log
Flash Engine exit code: 2
All updates failed.
Flash complete [7]

I have a dual port Emulex LPe1105-HP

root@suse11:~ # systool -av -c fc_host | grep “Class Device =”
Class Device = “host0”
Class Device = “host1”

root@suse11:~ # systool -c scsi_host -v -d host0 | grep modeldesc
modeldesc = “HP BLc Emulex LPe1105-HP FC Mezz Option Kit”
root@suse11:~ # systool -c scsi_host -v -d host1 | grep modeldesc
modeldesc = “HP BLc Emulex LPe1105-HP FC Mezz Option Kit”

Reviewing the log file, it seems that this server already had the firmware flashed

root@suse11:~ # cat /var/cpq/Component.log
=================================================================

Installation started: Wed Oct 7 10:57:33 BRT 2015

Installing: CP022701.scexe
Version: 2014.04.01a
Description: HP Firmware Flash for Emulex Fibre Channel HBAs and CNAs

—————————————————
Loading references…done.
Performing discovery…
Preparing the Emulex Kit…done
Performing flash — DO NOT INTERRUPT!
Executing flash engine: ./elxflashStandalone*/lx/elxflash.sh /sysfs /update 1>> /var/cpq/Component.log 2>> /var/cpq/Component.log 3>> /var/cpq/Component.log

Wed Oct 7 10:58:12 2015
HBA=LPe1105, Port Type=FC, WWN=10:00:00:00:C9:9A:35:8E,
Update=Firmware, Image=ZF282X4.ALL, New=282X4, Old=280A4, Status=Success

Wed Oct 7 10:58:33 2015
HBA=LPe1105, Port Type=FC, WWN=10:00:00:00:C9:9A:35:8E,
Update=Boot Code, Image=ZU610A14.PRG, New=610A14, Old=603A7, Status=Success

Wed Oct 7 10:59:11 2015
HBA=LPe1105, Port Type=FC, WWN=10:00:00:00:C9:9A:35:8F,
Update=Firmware, Image=ZF282X4.ALL, New=282X4, Old=280A4, Status=Success

Wed Oct 7 10:59:32 2015
HBA=LPe1105, Port Type=FC, WWN=10:00:00:00:C9:9A:35:8F,
Update=Boot Code, Image=ZU610A14.PRG, New=610A14, Old=603A7, Status=Success

elxflash: All required updates succeeded – Return Code=0

Flash Engine exit code: 0
All updates were successful. Reboot needed.
Flash complete [1]

Return code: 1

Installation complete: Wed Oct 7 10:59:32 BRT 2015

=================================================================

—————————————————
Loading references…done.
Performing discovery…
Preparing the Emulex Kit…done
Performing flash — DO NOT INTERRUPT!
Executing flash engine: ./elxflashStandalone*/lx/elxflash.sh /fc /update 1>> /var/cpq/Component.log 2>> /var/cpq/Component.log 3>> /var/cpq/Component.log

elxflash: Error – all required updates failed – Return Code=2

Flash Engine exit code: 2
All updates failed.
Flash complete [7]
—————————————————
Loading references…done.
Performing discovery…
Preparing the Emulex Kit…done
Performing flash — DO NOT INTERRUPT!
Executing flash engine: ./elxflashStandalone*/lx/elxflash.sh /fc /update 1>> /var/cpq/Component.log 2>> /var/cpq/Component.log 3>> /var/cpq/Component.log

elxflash: Error – all required updates failed – Return Code=2

Flash Engine exit code: 2
All updates failed.
Flash complete [7]