Tag: umount

Linux – umount: /filesystem: device is busy. No processes found using lsof and fuser

root@linux:~ # df -hP | grep SCR
/dev/mapper/vgSAPlocal-lv_sapmnt_SCR 15G 2.0G 12G 15% /sapmnt/SCR
scsscr:/export/sapmnt/SCR/exe 4.2G 3.5G 473M 89% /sapmnt/SCR/exe
scsscr:/export/sapmnt/SCR/global 4.2G 362M 3.6G 9% /sapmnt/SCR/global
scsscr:/export/sapmnt/SCR/profile 2.0G 3.0M 1.9G 1% /sapmnt/SCR/profile

I unmounted the filesystems under /sapmnt/SCR

root@linux:~ # umount /sapmnt/SCR/exe /sapmnt/SCR/global /sapmnt/SCR/profile
root@linux:~ #

But I was unable to unmount /sapmnt/SCR

root@linux:~ # umount /sapmnt/SCR
umount: /sapmnt/SCR: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))

After using fuser and lsof, it didn’t find any processes.
I was able to unmount it after restarting the autofs service

root@linux:~ # service autofs restart
Stopping automount: [ OK ]
Starting automount: [ OK ]

Advertisement

Linux EXT4-fs: error (device dm-156): ext4_lookup: deleted inode referenced: 1091357

Node : serviceguardnode2.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : minor
OM Server Time: 2016-12-22 18:22:32
Message : EXT4-fs: error (device dm-156): ext4_lookup: deleted inode referenced: 1091357
Msg Group : OS
Application : dmsg_mon
Object : EXT4
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Checking which device is complaining. dm-156 is /dev/vgWPJ/lv_orawp0

root@serviceguardnode2:/dev/mapper # ls -l | grep 156
lrwxrwxrwx. 1 root root 9 Dec 14 22:15 vgWPJ-lv_orawp0 -> ../dm-156

The filesystem is currently mounted

root@serviceguardnode2:/dev/mapper # mount | grep lv_orawp0
/dev/mapper/vgWPJ-lv_orawp0 on /oracle/WPJ type ext4 (rw,errors=remount-ro,data_err=abort,barrier=0)

And the logical volume is open

root@serviceguardnode2:~ # lvs vgWPJ
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_ora11264 vgWPJ -wi-ao—- 30.00g
lv_orawp0 vgWPJ -wi-ao—- 5.00g

This is a clustered environment and it is currently running on the other node

root@serviceguardnode2:/dev/mapper # cmviewcl | grep -i wpj
dbWPJ up running enabled serviceguardnode1

There is a Red Hat note referencing the error – “ext4_lookup: deleted inode referenced” errors in /var/log/messages in RHEL 6.

In clustered environments, which is the case, if the other node is mounting the filesystem, it will throw these errors in /var/log/messages

root@serviceguardnode2:~ # cmviewcl -v -p dbWPJ

PACKAGE STATUS STATE AUTO_RUN NODE
dbWPJ up running enabled serviceguardnode1

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 5 0 dbWPJmon
Subnet up 10.106.10.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled serviceguardnode1 (current)
Alternate up enabled serviceguardnode2

Dependency_Parameters:
DEPENDENCY_NAME NODE_NAME SATISFIED
dbWP0_dep serviceguardnode2 no
dbWP0_dep serviceguardnode1 yes

Other_Attributes:
ATTRIBUTE_NAME ATTRIBUTE_VALUE
Style modular
Priority no_priority

Checking the filesystems. I need to unmount /oracle/WPJ but first I need to umount everything under /oracle/WPJ otherwise it will show that /oracle/WPJ is busy

root@serviceguardnode2:~ # df -hP | grep WPJ
/dev/mapper/vgSAP-lv_WPJ_sys 93M 1.6M 87M 2% /usr/sap/WPJ/SYS
/dev/mapper/vgWPJ-lv_orawp0 4.4G 162M 4.0G 4% /oracle/WPJ
/dev/mapper/vgWPJ-lv_ora11264 27G 4.7G 21G 19% /oracle/WPJ/11204
/dev/mapper/vgWPJlog2-lv_origlogb 2.0G 423M 1.4G 23% /oracle/WPJ/origlogB
/dev/mapper/vgWPJlog2-lv_mirrloga 2.0G 404M 1.5G 22% /oracle/WPJ/mirrlogA
/dev/mapper/vgWPJlog1-lv_origloga 2.0G 423M 1.4G 23% /oracle/WPJ/origlogA
/dev/mapper/vgWPJlog1-lv_mirrlogb 2.0G 404M 1.5G 22% /oracle/WPJ/mirrlogB
/dev/mapper/vgWPJdata-lv_sapdata4 75G 21G 55G 28% /oracle/WPJ/sapdata4
/dev/mapper/vgWPJdata-lv_sapdata3 75G 79M 75G 1% /oracle/WPJ/sapdata3
/dev/mapper/vgWPJdata-lv_sapdata2 75G 7.3G 68G 10% /oracle/WPJ/sapdata2
/dev/mapper/vgWPJdata-lv_sapdata1 75G 1.1G 74G 2% /oracle/WPJ/sapdata1
/dev/mapper/vgWPJoraarch-lv_oraarch 20G 234M 19G 2% /oracle/WPJ/oraarch
scsWPJ:/export/sapmnt/WPJ/profile 4.4G 4.0M 4.1G 1% /sapmnt/WPJ/profile
scsWPJ:/export/sapmnt/WPJ/exe 4.4G 2.5G 1.7G 61% /sapmnt/WPJ/exe

Umounting /oracle/WPJ

root@serviceguardnode2:~ # umount /oracle/WPJ/11204
root@serviceguardnode2:~ # umount /oracle/WPJ/origlogB
root@serviceguardnode2:~ # umount /oracle/WPJ/mirrlogA
root@serviceguardnode2:~ # umount /oracle/WPJ/origlogA
root@serviceguardnode2:~ # umount /oracle/WPJ/mirrlogB
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata4
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata3
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata2
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata1
root@serviceguardnode2:~ # umount /oracle/WPJ/oraarch
root@serviceguardnode2:~ # umount /oracle/WPJ

Samba share without permission. Directory showing as d———

Server is mounting network share being exported using CIFS

root@linux:~ # df -hP /arq/avf/ROT_EFC
Filesystem Size Used Avail Use% Mounted on
//172.20.1.2/Operacao_ROT_EFC$ 43G 40G 3.2G 93% /arq/avf/ROT_EFC

root@linux:~ # mount | grep ROT_EFC
//172.20.1.2/Operacao_ROT_EFC$ on /arq/avf/ROT_EFC type cifs (rw)

Information about the filesystem on /etc/fstab

root@linux:~ # grep Operacao_ROT_EFC /etc/fstab
//172.20.1.2/Operacao_ROT_EFC$ /arq/avf/ROT_EFC cifs _netdev,user=s-ad-USER1468,pass=userpassword,uid=21376,gid=889,file_mode=0775,dir_mode=0775,domain=setaoffice,cifsacl

There is no permission and it can’t be changed by the Linux server brqsb1valeas890

root@linux:~ # ls -ld /arq/avf/ROT_EFVM /arq/avf/ROT_EFC
d——— 7 user1468 admweb 0 Sep 15 15:53 /arq/avf/ROT_EFC

root@linux:~ # chmod 775 /arq/avf/ROT_EFC
chmod: changing permissions of `/arq/avf/ROT_EFC’: Permission denied

No problem mounting manually

mount -t cifs //172.20.1.2/Operacao_ROT_EFC$ /arq/avf/ROT_EFC -o “username=s-ad-USER1468,domain=setaoffice,uid=21376,gid=889,file_mode=0775,dir_mode=0775”

Rewrote entry in /etc/fstab. Must have been a hidden character

root@linux:~ # umount /arq/avf/ROT_EFC
root@linux:~ # mount /arq/avf/ROT_EFC
root@linux:~ # ls -dl /arq/avf/ROT_EFC
drwxrwxr-x 7 user1468 admweb 0 Sep 15 15:53 /arq/avf/ROT_EFC

Problems with a NFS share coming from a Linux server and mounted on Linux client

Running df or ls in a NFS share it simply hangs.

See if df or ls hangs. Run strace df -h or strace ls -l and see if it finishes

root@linux:~ # strace df -h

root@linux:~ # strace ls -l

statfs(“/mnt/NFS”,

Umount with -f or -l

-f, –force
Force an unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.)

Note that this option does not guarantee that umount command does not hang. It’s strongly recommended to use absolute paths without symlinks to avoid unwanted readlink and stat syscalls on unreachable NFS in umount.
-l, –lazy
Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore. (Requires kernel 2.4.11 or later.)

Problems with a NFS share coming from a Linux server and mounted on Linux client

Running df or ls in a NFS share it simply hangs.

See if df or ls hangs. Run strace df -h or strace ls -l and see if it finishes

root@linux:~ # strace df -h

root@linux:~ # strace ls -l

statfs(“/mnt/NFS”,

Try to umount with -f or -l

-f, –force
Force an unmount (in case of an unreachable NFS system). (Requires kernel 2.1.116 or later.)

Note that this option does not guarantee that umount command does not hang. It’s strongly recommended to use absolute paths without symlinks to avoid unwanted readlink and stat syscalls on unreachable NFS in umount.

-l, –lazy
Lazy unmount. Detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore. (Requires kernel 2.4.11 or later.)

Linux – command ls or df hangs on /

If you run ls or df -h, these commands will appear that hung

Check if you have NFS shares mounted

root@linux:~ # mount | grep nfs
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
mdmMPC:/export/sapmnt/MPC on /sapmnt/MPC type nfs (rw,nfsvers=3,proto=tcp,noac,soft,sloppy,addr=10.106.10.118)
linuxnfs25:/oracle/HP0/sapdata1/DUMP on /dump type nfs (rw,addr=142.40.81.32)
nfshp0:/export/sapmnt/HP0/exe on /sapmnt/HP0/exe type nfs (rw,nfsvers=3,proto=udp,noac,soft,sloppy,addr=10.106.10.28)
nfshp0:/export/sapmnt/HP0/profile on /sapmnt/HP0/profile type nfs (rw,nfsvers=3,proto=udp,noac,soft,sloppy,addr=10.106.10.28)

I have this share that the server was turned off. So I tried to umount the share

root@linux:~ # umount /dump
umount.nfs: /dump: device is busy
umount.nfs: /dump: device is busy

And even forcing but no luck

root@linux:~ # umount -f /dump
umount2: Device or resource busy
umount.nfs: /dump: device is busy
umount2: Device or resource busy
umount.nfs: /dump: device is busy

umount with -l to do a lazy unmount

root@linux:~ # umount -l /dump
root@linux:~ # df -h /dump
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lv_root
7.8G 818M 6.6G 11% /

ORA-00206: error in writing (block 42, # blocks 1) of control file

DBA notified me of this error

Wed Nov 25 13:46:11 BRST 2015
Errors in file /usr/software/oracle/admin/orarvt015/bdump/rvt015_lgwr_2388.trc:
ORA-00206: error in writing (block 42, # blocks 1) of control file
ORA-00202: control file: ‘/usr/oradata/orarvt015/control2/control02.ctl’
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error

Umount the filesystem

root@linux:~ # umount /usr/oradata/orarvt015

I ran fsck

root@linux:~ # fsck -t ext3 /dev/mapper/oradatavg-dat.orarvt015new
fsck 1.38 (30-Jun-2005)
e2fsck 1.38 (30-Jun-2005)
/dev/mapper/oradatavg-dat.orarvt015new: recovering journal
/dev/mapper/oradatavg-dat.orarvt015new: clean, 137/209715200 files, 318872229/419428352 blocks

Then I mounted the filesystem again

root@linux:~ # mount /usr/oradata/orarvt015

Showing filesystem size

root@linux:~ # df -h /usr/oradata/orarvt015
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/oradatavg-dat.orarvt015new
1.6T 1.2T 304G 80% /usr/oradata/orarvt015

It solved the problem