Tag: dmsetup

Disk removed on Linux and then LVM commands giving error: read failed after 0 of 4096 at 0: Input/output error

Some LUNs were removed from the server and now LVM commands give error messages

root@linux:~# pvs
/dev/mapper/350002ac4f691374a: read failed after 0 of 4096 at 0: Input/output error
/dev/mapper/350002ac4f691374a: read failed after 0 of 4096 at 107374116864: Input/output error
/dev/mapper/350002ac4f691374a: read failed after 0 of 4096 at 107374174208: Input/output error
/dev/mapper/350002ac4f691374a: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_ASCS: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_ASCS: read failed after 0 of 4096 at 10737352704: Input/output error
/dev/vgHP0ascs/lv_ASCS: read failed after 0 of 4096 at 10737410048: Input/output error
/dev/vgHP0ascs/lv_ASCS: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_NOVELL_RemoteLoader: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_NOVELL_RemoteLoader: read failed after 0 of 4096 at 117374976: Input/output error
/dev/vgHP0ascs/lv_NOVELL_RemoteLoader: read failed after 0 of 4096 at 117432320: Input/output error
/dev/vgHP0ascs/lv_NOVELL_RemoteLoader: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_sapmnt_global: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_sapmnt_global: read failed after 0 of 4096 at 10737352704: Input/output error
/dev/vgHP0ascs/lv_sapmnt_global: read failed after 0 of 4096 at 10737410048: Input/output error
/dev/vgHP0ascs/lv_sapmnt_global: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_sapmnt_profile: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_sapmnt_profile: read failed after 0 of 4096 at 2147418112: Input/output error
/dev/vgHP0ascs/lv_sapmnt_profile: read failed after 0 of 4096 at 2147475456: Input/output error
/dev/vgHP0ascs/lv_sapmnt_profile: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_usrsapHP0: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_usrsapHP0: read failed after 0 of 4096 at 7516127232: Input/output error
/dev/vgHP0ascs/lv_usrsapHP0: read failed after 0 of 4096 at 7516184576: Input/output error
/dev/vgHP0ascs/lv_usrsapHP0: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_sapmnt: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_sapmnt: read failed after 0 of 4096 at 20971454464: Input/output error
/dev/vgHP0ascs/lv_sapmnt: read failed after 0 of 4096 at 20971511808: Input/output error
/dev/vgHP0ascs/lv_sapmnt: read failed after 0 of 4096 at 4096: Input/output error
/dev/vgHP0ascs/lv_sapmnt_exe: read failed after 0 of 4096 at 0: Input/output error
/dev/vgHP0ascs/lv_sapmnt_exe: read failed after 0 of 4096 at 5368643584: Input/output error
/dev/vgHP0ascs/lv_sapmnt_exe: read failed after 0 of 4096 at 5368700928: Input/output error
/dev/vgHP0ascs/lv_sapmnt_exe: read failed after 0 of 4096 at 4096: Input/output error
PV VG Fmt Attr PSize PFree
/dev/sda2 vgroot lvm2 a– 99.50g 38.74g

Check with the command dmsetup the device that needs to be removed and then remove it. I do it from bottom to top

root@linux:~# dmsetup ls | grep lv_sapmnt_exe
vgHP0ascs-lv_sapmnt_exe (253:21)
root@linux:~# dmsetup remove vgHP0ascs-lv_sapmnt_exe

root@linux:~# dmsetup ls | grep lv_sapmnt
vgHP0ascs-lv_sapmnt_profile (253:18)
vgHP0ascs-lv_sapmnt_global (253:17)
vgHP0ascs-lv_sapmnt (253:20)
root@linux:~# dmsetup remove vgHP0ascs-lv_sapmnt

root@linux:~# dmsetup ls | grep lv_usrsapHP0
vgHP0ascs-lv_usrsapHP0 (253:19)
root@linux:~# dmsetup remove vgHP0ascs-lv_usrsapHP0

root@linux:~# dmsetup ls | grep lv_sapmnt_profile
vgHP0ascs-lv_sapmnt_profile (253:18)
root@linux:~# dmsetup remove vgHP0ascs-lv_sapmnt_profile

root@linux:~# dmsetup ls | grep lv_sapmnt_global
vgHP0ascs-lv_sapmnt_global (253:17)
root@linux:~# dmsetup remove vgHP0ascs-lv_sapmnt_global

root@linux:~# dmsetup ls | grep lv_NOVELL_RemoteLoader
vgHP0ascs-lv_NOVELL_RemoteLoader (253:16)
root@linux:~# dmsetup remove vgHP0ascs-lv_NOVELL_RemoteLoader

root@linux:~# dmsetup ls | grep lv_ASCS
vgHP0ascs-lv_ASCS (253:15)
root@linux:~# dmsetup remove vgHP0ascs-lv_ASCS

root@linux:~# dmsetup ls | grep 350002ac4f691374a
350002ac4f691374a (253:14)
root@linux:~# dmsetup remove 350002ac4f691374a

Then after issuing pvs, it won’t show error messages anymore

root@linux:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vgroot lvm2 a– 99.50g 38.74g

LVM – Disk removed and appears message read failed after 0 of 4096 at 4096 or similar

In this server when I ran pvs it was showing a lot of input/output errors

root@linux:~ # pvs
/dev/sdq: read failed after 0 of 4096 at 0: Input/output error
/dev/sdq1: read failed after 0 of 2048 at 0: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 80530571264: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 80530628608: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 0: Input/output error
/dev/mpath/disk2: read failed after 0 of 4096 at 4096: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 80525328384: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 80525447168: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 0: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 512 at 4096: Input/output error
/dev/mpath/disk2p1: read failed after 0 of 2048 at 0: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 21474770944: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 21474828288: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 0: Input/output error
/dev/devvg/u01lv: read failed after 0 of 4096 at 4096: Input/output error
/dev/sde: read failed after 0 of 4096 at 0: Input/output error
/dev/sde1: read failed after 0 of 2048 at 0: Input/output error
/dev/sdw: read failed after 0 of 4096 at 0: Input/output error
/dev/sdw1: read failed after 0 of 2048 at 0: Input/output error
/dev/sdk: read failed after 0 of 4096 at 0: Input/output error

A disk was removed and it was not removed cleanly

root@linux:~ # multipath -ll disk2
sde: checker msg is “tur checker reports path is down”
sdk: checker msg is “tur checker reports path is down”
sdq: checker msg is “tur checker reports path is down”
sdw: checker msg is “tur checker reports path is down”
disk2 (360000000000000000000000000000000) dm-25 HP,P2000 G3 FC
[size=75G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][enabled]
\_ 1:0:0:5 sde 8:64 [failed][faulty]
\_ 1:0:1:5 sdk 8:160 [failed][faulty]
\_ 2:0:0:5 sdq 65:0 [failed][faulty]
\_ 2:0:1:5 sdw 65:96 [failed][faulty]

Removed the block devices

root@linux:~ # echo 1 > /sys/block/sde/device/delete
root@linux:~ # echo 1 > /sys/block/sdk/device/delete
root@linux:~ # echo 1 > /sys/block/sdq/device/delete
root@linux:~ # echo 1 > /sys/block/sdw/device/delete

I was able to remove the device that device-mapper used but I was not able to remove the device with the partition

root@linux:~ # dmsetup remove disk2
root@linux:~ # dmsetup remove disk2p1
device-mapper: remove ioctl failed: Device or resource busy
Command failed

I had a filesystem using the device /dev/devvg/u01lv so I had to remove it before I was able to remove the device with the partition

root@linux:~ # dmsetup remove devvg-u01lv

root@linux:~ # dmsetup remove disk2p1

device-mapper: create ioctl failed: Device or resource busy

I was trying to create the device file partition but I was receiving an error message

root@linux:~ # kpartx -a /dev/mapper/mpathm
device-mapper: create ioctl failed: Device or resource busy
create/reload failed on mpathm1

The disk already had a partition. Let’s use dmsetup to see what we have

root@linux:~ # dmsetup ls | grep mpathm
mpathm-part1 (253, 38)
mpathm (253, 32)

We already have a mpathm partition registered so we need to remove

root@linux:~ # dmsetup remove mpathm-part1

Then retry

root@linux:~ # kpartx -a /dev/mapper/mpathm
root@linux:~ #