Advertisements

Tag Archives: cmviewcl

Linux EXT4-fs: error (device dm-156): ext4_lookup: deleted inode referenced: 1091357

Node : serviceguardnode2.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : minor
OM Server Time: 2016-12-22 18:22:32
Message : EXT4-fs: error (device dm-156): ext4_lookup: deleted inode referenced: 1091357
Msg Group : OS
Application : dmsg_mon
Object : EXT4
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Checking which device is complaining. dm-156 is /dev/vgWPJ/lv_orawp0

root@serviceguardnode2:/dev/mapper # ls -l | grep 156
lrwxrwxrwx. 1 root root 9 Dec 14 22:15 vgWPJ-lv_orawp0 -> ../dm-156

The filesystem is currently mounted

root@serviceguardnode2:/dev/mapper # mount | grep lv_orawp0
/dev/mapper/vgWPJ-lv_orawp0 on /oracle/WPJ type ext4 (rw,errors=remount-ro,data_err=abort,barrier=0)

And the logical volume is open

root@serviceguardnode2:~ # lvs vgWPJ
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
lv_ora11264 vgWPJ -wi-ao—- 30.00g
lv_orawp0 vgWPJ -wi-ao—- 5.00g

This is a clustered environment and it is currently running on the other node

root@serviceguardnode2:/dev/mapper # cmviewcl | grep -i wpj
dbWPJ up running enabled serviceguardnode1

There is a Red Hat note referencing the error – “ext4_lookup: deleted inode referenced” errors in /var/log/messages in RHEL 6.

In clustered environments, which is the case, if the other node is mounting the filesystem, it will throw these errors in /var/log/messages

root@serviceguardnode2:~ # cmviewcl -v -p dbWPJ

PACKAGE STATUS STATE AUTO_RUN NODE
dbWPJ up running enabled serviceguardnode1

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 5 0 dbWPJmon
Subnet up 10.106.10.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled serviceguardnode1 (current)
Alternate up enabled serviceguardnode2

Dependency_Parameters:
DEPENDENCY_NAME NODE_NAME SATISFIED
dbWP0_dep serviceguardnode2 no
dbWP0_dep serviceguardnode1 yes

Other_Attributes:
ATTRIBUTE_NAME ATTRIBUTE_VALUE
Style modular
Priority no_priority

Checking the filesystems. I need to unmount /oracle/WPJ but first I need to umount everything under /oracle/WPJ otherwise it will show that /oracle/WPJ is busy

root@serviceguardnode2:~ # df -hP | grep WPJ
/dev/mapper/vgSAP-lv_WPJ_sys 93M 1.6M 87M 2% /usr/sap/WPJ/SYS
/dev/mapper/vgWPJ-lv_orawp0 4.4G 162M 4.0G 4% /oracle/WPJ
/dev/mapper/vgWPJ-lv_ora11264 27G 4.7G 21G 19% /oracle/WPJ/11204
/dev/mapper/vgWPJlog2-lv_origlogb 2.0G 423M 1.4G 23% /oracle/WPJ/origlogB
/dev/mapper/vgWPJlog2-lv_mirrloga 2.0G 404M 1.5G 22% /oracle/WPJ/mirrlogA
/dev/mapper/vgWPJlog1-lv_origloga 2.0G 423M 1.4G 23% /oracle/WPJ/origlogA
/dev/mapper/vgWPJlog1-lv_mirrlogb 2.0G 404M 1.5G 22% /oracle/WPJ/mirrlogB
/dev/mapper/vgWPJdata-lv_sapdata4 75G 21G 55G 28% /oracle/WPJ/sapdata4
/dev/mapper/vgWPJdata-lv_sapdata3 75G 79M 75G 1% /oracle/WPJ/sapdata3
/dev/mapper/vgWPJdata-lv_sapdata2 75G 7.3G 68G 10% /oracle/WPJ/sapdata2
/dev/mapper/vgWPJdata-lv_sapdata1 75G 1.1G 74G 2% /oracle/WPJ/sapdata1
/dev/mapper/vgWPJoraarch-lv_oraarch 20G 234M 19G 2% /oracle/WPJ/oraarch
scsWPJ:/export/sapmnt/WPJ/profile 4.4G 4.0M 4.1G 1% /sapmnt/WPJ/profile
scsWPJ:/export/sapmnt/WPJ/exe 4.4G 2.5G 1.7G 61% /sapmnt/WPJ/exe

Umounting /oracle/WPJ

root@serviceguardnode2:~ # umount /oracle/WPJ/11204
root@serviceguardnode2:~ # umount /oracle/WPJ/origlogB
root@serviceguardnode2:~ # umount /oracle/WPJ/mirrlogA
root@serviceguardnode2:~ # umount /oracle/WPJ/origlogA
root@serviceguardnode2:~ # umount /oracle/WPJ/mirrlogB
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata4
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata3
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata2
root@serviceguardnode2:~ # umount /oracle/WPJ/sapdata1
root@serviceguardnode2:~ # umount /oracle/WPJ/oraarch
root@serviceguardnode2:~ # umount /oracle/WPJ

Advertisements

cmrunpkg: Unable to start some package or package instances.

Checking Serviceguard cluster packages

root@linux01:~ # cmviewcl

CLUSTER STATUS
cluster_virtual_scc_004 up

NODE STATUS STATE
linux01 up running
linux02 up running

PACKAGE STATUS STATE AUTO_RUN NODE
infraWP0 up running enabled linux02
ascsWP0 up running enabled linux02
wdpWP0 up running enabled linux02
infraSCP up running enabled linux02
scsSCP up running enabled linux02

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
wdpSCP down failed enabled unowned

When trying to start the cluster package wdpSCP on server linux02 it fails because it says the node is not eligible

root@linux01:~ # cmrunpkg -n linux02 wdpSCP
Checking for license………
Found Valid Advanced License
Number of Advanced licenses:1
Unable to run package wdpSCP on node linux02. Node is not eligible.
cmrunpkg: Unable to start some package or package instances.

Checking detailed statistics about the cluster package wdpSCP says that it is disabled to run on node linux02

root@linux01:~ # cmviewcl -v -p wdpSCP

UNOWNED_PACKAGES

PACKAGE STATUS STATE AUTO_RUN NODE
wdpSCP down failed enabled unowned

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS NODE_NAME NAME

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled linux01
Alternate up disabled linux02

Dependency_Parameters:
DEPENDENCY_NAME NODE_NAME SATISFIED
infraSCP_dep linux01 no
infraSCP_dep linux02 yes

Other_Attributes:
ATTRIBUTE_NAME ATTRIBUTE_VALUE
Style modular
Priority no_priority

Enable the cluster package to run on node linux02

root@linux01:~ # cmmodpkg -e -v -n linux02 wdpSCP
Checking for license………
Found Valid Advanced License
Number of Advanced licenses:1
Enabling node linux02 for switching of package wdpSCP
Successfully enabled package wdpSCP to run on node linux02
cmmodpkg: Completed successfully on all packages specified

UXMON: ServiceGuard: Package(s) are running on ADOPTIVE node(s): infraHP0 (linux02)

UXMON: ServiceGuard: Package(s) are running on ADOPTIVE node(s): infraHP0 (linux02)

Node : linux01.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2016-02-22 14:54:56
Message : UXMON: ServiceGuard: Package(s) are running on ADOPTIVE node(s): infraHP0 (linux02)
Msg Group : OS
Application : sgmon
Object : cmviewcl
Event Type :
not_found

Instance Name :
not_found

Instruction : Check with cmviewcl -v;

If package was down the customer has to be informed about the downtime.

Do not close this case before it is resolved.
As long as this EWM-case is not resolved or closed, monitoring is disabled
In any case inform technical lead!

This ticket is about the cluster package that it is running on alternate node.

root@linux:~ # cp -p /var/opt/OV/bin/instrumentation/sg_mon.cfg /var/opt/OV/conf/OpC/sg_mon.cfg
root@linux:~ # vi /var/opt/OV/conf/OpC/sg_mon.cfg
# PKG[0]=XYZ Package name 1
# PKG_NODE[0]=ABC Primary node on which the package must run
# Define as * will disable running on adoptive node check
# PKG_SWTCH[0]=1 Set to 1 if Package_switching should be ENABLED
# Set to 0 if Package_switching must not be ENABLED

In our case, it was disabled the node check for the package

root@linux:~ # vi /var/opt/OV/conf/OpC/sg_mon.cfg
PKG[0]=infraHP0; PKG_NODE[0]=*; PKG_SWTCH[0]=0

%d bloggers like this: