Advertisements

Category Archives: Solaris

UXMON: Service /application/pkg/zones-proxy-client status is maintenance, check with svcs -xv

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : localzone.setaoffice.com
Node Type : Sun SPARC (HTTPS)
Severity : minor
OM Server Time: 2017-09-24 22:32:27
Message : UXMON: Service /application/pkg/zones-proxy-client status is maintenance, check with svcs -xv
Msg Group : OS
Application : svcsmon
Object : svcs
Event Type :
not_found

Instance Name :
not_found

Instruction : The svcsmon has detected solaris service status

Please, for details, browse the /var/opt/OV/log/OpC/svcs_mon.log
The configuration file uses to be /var/opt/OV/conf/OpC/svcs_mon.cfg

On global zone

root@globalzone:~ # pkg publisher
PUBLISHER TYPE STATUS P LOCATION
solaris origin online F file:///net/192.168.252.12/export/IPS-repos/solaris11/repo/
solaris origin online F file:///var/scmuidrs/idr2142.1.p5p/
solaris origin online F file:///var/scmuidrs/idr2160.2.p5p/
solaris origin online F file:///var/scmuidrs/idr2193.2.p5p/
solaris origin online F file:///var/scmuidrs/idr2194.1.p5p/
solaris origin online F file:///var/scmuidrs/idr2238.1.p5p/
exa-family origin online F file:///net/192.168.252.12/export/IPS-repos/exafamily/repo/
ha-cluster origin online F file:///net/192.168.252.12/export/IPS-repos/osc4/repo/

root@globalzone:~ # svcadm disable zones-proxyd system-repository; svcadm enable system-repository zones-proxyd; sleep 30
root@globalzone:~ #

On local zone

root@localzone:~# svcs -a | grep /application/pkg/zones-proxy-client
maintenance 8:08:15 svc:/application/pkg/zones-proxy-client:default

root@localzone:~# svcadm disable svc:/application/pkg/zones-proxy-client:default

root@localzone:~# svcadm enable svc:/application/pkg/zones-proxy-client:default

root@localzone:~# svcs -a | grep /application/pkg/zones-proxy-client
online 14:09:45 svc:/application/pkg/zones-proxy-client:default

Advertisements

Oracle SMF Oracle Configuration Manager (OCM) svc:/system/ocm:default

Listing status of ocm service

root@solaris:~ # svcs svc:/system/ocm:default
STATE STIME FMRI
disabled Jul_17 svc:/system/ocm:default

Listing SMF

root@solaris:~ # svcs -l svc:/system/ocm:default
fmri svc:/system/ocm:default
name Oracle Configuration Manager (OCM)
enabled false
state disabled
next_state none
state_time Mon Jul 17 04:48:06 2017
logfile /var/svc/log/system-ocm:default.log
restarter svc:/system/svc/restarter:default
contract_id
manifest /etc/svc/profile/generic.xml
manifest /lib/svc/manifest/system/ocm.xml
dependency require_all/none svc:/milestone/multi-user-server:default (online)
dependency require_all/error svc:/milestone/network:default (online)
dependency require_all/none svc:/system/cryptosvc (online)

Oracle Configuration Manager is used to collect client configuration information and upload it to the Oracle repository
When enabling OCM, it goes to maintenance mode

root@solaris:~ # svcadm enable svc:/system/ocm:default

root@solaris:~ # svcs -v svc:/system/ocm:default
STATE NSTATE STIME CTID FMRI
maintenance – 10:58:25 1081435 svc:/system/ocm:default

There are two errors in this installation. There is no proxy setup and it is missing user ocm

root@solaris:~ # cat /var/svc/log/system-ocm:default.log
[ Aug 28 10:58:09 Enabled. ]
[ Aug 28 10:58:09 Executing start method (“/lib/svc/method/svc-ocm start”). ]
/lib/svc/method/svc-ocm: starting…
OCM not registered
Collector running in connected mode
Begin anonymous registration…
Starting response file generation…
Can not create response file: Unknown Host: ccr.oracle.com: unknown error
Failed to create response file…
Failed to generate anonymous response file…
Unable to contact ccr.oracle.com. Please set your system proxy
in order to allow this system to contact Oracle for better
serviceability. See the configCCR(1M) manual page on home to set
the proxy server for Oracle Configuration Manager.

svc:/system/ocm:default has been temporarily disabled.

[ Aug 28 10:58:20 Method “start” exited with status 0. ]
[ Aug 28 10:58:20 Stopping because service disabled. ]
[ Aug 28 10:58:20 Executing stop method (“/lib/svc/method/svc-ocm stop”). ]
Stopping scheduler…
su: Unknown id: ocm
[ Aug 28 10:58:25 Method “stop” exited with status 95. ]

Solaris – UXMON: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major

UXMON: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major

Node : solaris.setaoffice.com
Node Type : Sun SPARC (HTTPS)
Severity : major
OM Server Time: 2017-08-12 10:27:31
Message : UXMON: [ID 377184 daemon.error] SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major
Msg Group : OS
Application : SOL_mon
Object : FMT
Event Type :
not_found

Instance Name :
not_found

Instruction : “The Fault Management agent has identified a HW or OS related problem with the severity presented by the ticket.
The problem(s) can be viewed and managed with the command – fmdump
To get a better understanding of the problem and on how to resolve it, locate the event that generated
the ticket in the syslog file /var/adm/messages, a URL will be found (http://sun.com/msg/xxx-nnnn-yy),
follow the link using your Oracle portal account for instructions.”

After running fmadm faulty, we see that there is a problem with a zpool. Run zpool status -x and then we see pool prd027_software is having problems

root@solaris:~ # zpool status prd027_software
pool: prd027_software
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
Run ‘zpool status -v’ to see device specific details.
see: http://support.oracle.com/msg/ZFS-8000-8A
scan: none requested
config:

NAME STATE READ WRITE CKSUM
prd027_software ONLINE 0 0 14.7K
c0t600507680191818C1000000000000BE9d0 ONLINE 0 0 0
c0t600507680191818C1000000000000BEAd0 ONLINE 0 0 0
c0t600507680191818C1000000000000BEBd0 ONLINE 0 0 0
c0t600507680191818C1000000000000BECd0 ONLINE 0 0 0

errors: 3 data errors, use ‘-v’ for a list

Run zpool scrub prd027_software

root@solaris:~ # zpool scrub prd027_software

root@solaris:~ # zpool status -xv
pool: prd027_software
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://support.oracle.com/msg/ZFS-8000-8A
scan: scrub in progress since Wed Dec 31 21:00:00 1969
50.7M scanned out of 1.08T at 25.3M/s, 12h25m to go
0 repaired, 0.00% done
config:

NAME STATE READ WRITE CKSUM
prd027_software ONLINE 0 0 14.7K
c0t600507680191818C1000000000000BE9d0 ONLINE 0 0 0
c0t600507680191818C1000000000000BEAd0 ONLINE 0 0 0
c0t600507680191818C1000000000000BEBd0 ONLINE 0 0 0
c0t600507680191818C1000000000000BECd0 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

/zones/prd027/root/usr/software/best1/Patrol3/Solaris-2-10-sparc-64/best1/7.4.00/bgs/monitor/log/prd027-bgsagent_6767.als
prd027_software/software027:<0x198c996>
prd027_software/software027:<0x198de4e>

After the pool is scanned, check if there is still a problem

root@solaris:~ # zpool status -xv
all pools are healthy

Repairing fmadm entries

root@solaris:~ # fmadm faulty|grep “Aug”
Aug 12 11:23:22 82fe93a5-8120-657b-9e61-e33252b84d30 ZFS-8000-D3 Major
Aug 12 11:22:01 74c61e33-7c56-4aca-d707-a32ce06a9bd8 ZFS-8000-CS Major

root@solaris:~ # fmadm repair 82fe93a5-8120-657b-9e61-e33252b84d30
fmadm: recorded repair to 82fe93a5-8120-657b-9e61-e33252b84d30

root@solaris:~ # fmadm repair 74c61e33-7c56-4aca-d707-a32ce06a9bd8
fmadm: recorded repair to 74c61e33-7c56-4aca-d707-a32ce06a9bd8

root@solaris:~ # fmadm faulty
root@solaris:~ #

Solaris 10: passwd: password is based on a reversed dictionary word.

root@solaris:~ # passwd emerson
New Password:
passwd: password is based on a reversed dictionary word.

Please try again
New Password:

Edit file /etc/default/passwd and comment the files containing DICTIONDBDIR and DICTIONLIST

root@solaris:~ # vi /etc/default/passwd
#DICTIONDBDIR=/var/passwd
#DICTIONLIST=/usr/share/lib/dict/words

Checking serial number of Oracle SPARC T5-8

Connect to ILOM

emerson@linux:~ $ ssh root@172.23.99.70
Password:

Oracle(R) Integrated Lights Out Manager

Version 3.2.5.6.b r103360

Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: ssccn1-sp

-> show /System

/System
Targets:
Open_Problems (1)
CPU_Modules
Processors
Memory
Power
Cooling
Storage
Networking
PCI_Devices
Firmware
Log

Properties:
health = Service Required
health_details = PM0 (Processor Module 0) is faulty. Type ‘show /System/Open_Problems’ for details.
open_problems_count = 1
type = Rack Mount
model = SuperCluster T5-8
qpart_id = Q9527
part_number = SuperCluster T5-8
serial_number = AK00300268
component_model = SPARC T5-8
component_part_number = 7087535
component_serial_number = SP00386386
system_identifier = Oracle SuperCluster T5-8 SP00386386
system_fw_version = Sun System Firmware 9.5.1.b 2015/10/01 16:33
primary_operating_system = Oracle Solaris 11.3 SPARC
primary_operating_system_detail = –
host_primary_mac_address = 00:10:e0:76:92:de
ilom_address = 172.23.99.70
ilom_mac_address = 00:10:E0:76:92:E7
locator_indicator = Off
power_state = On
actual_power_consumption = 3958 watts
action = (none)

Commands:
cd
reset
set
show
start
stop

Another way is to locate the manufacturing sticker on the front of the server or on the sticker on the side of the server

Solaris 9 Branded Zone was not starting ftp when running kill -HUP

I have a Solaris 9 Branded Zone

root@solaris9:/ # uname -a
SunOS solaris9 5.9 Generic_Virtual sun4v sparc sun4v

Configured to run FTP

root@solaris9:/ # grep ftp /etc/inetd.conf
# ftp telnet shell login exec tftp finger printer
# TFTPD – tftp server (primarily used for booting)
#tftp dgram udp6 wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot
ftp stream tcp6 nowait root /usr/sbin/in.ftpd in.ftpd -l

But it was not working

root@solaris9:/ # ps -ef | grep ftp
root 10137 13230 0 13:31:28 pts/4 0:00 grep ftp

root@solaris9:/ # ps -ef | grep inet
root 12579 13230 0 13:31:34 pts/4 0:00 grep inet
root 1325 12833 0 Mar 12 ? 0:00 /usr/sbin/inetd -s start

Tried to kill -HUP but still not working

root@solaris9:/ # kill -HUP 1325

root@solaris9:/ # netstat -an | grep 21 | grep LISTEN
142.40.236.158.1521 *.* 0 0 1048576 0 LISTEN
142.40.236.10.1521 *.* 0 0 1048576 0 LISTEN

Stopped and started inetsvc

root@solaris9:/ # /etc/init.d/inetsvc stop
root@solaris9:/ # /etc/init.d/inetsvc start

root@solaris9:/ # ps -ef | grep inet
root 12098 12833 0 13:49:02 ? 0:00 /usr/sbin/inetd -s
root 15358 3734 0 13:49:05 pts/4 0:00 grep inet

FTP working again

root@solaris9:/ # netstat -an | grep 21 | grep LISTEN
142.40.236.158.1521 *.* 0 0 1048576 0 LISTEN
142.40.236.10.1521 *.* 0 0 1048576 0 LISTEN
*.21 *.* 0 0 1048576 0 LISTEN
*.21 *.* 0 0 1048576 0 LISTEN

UXMON: SSHD Daemon is not running or not doing it properly, please check

Node : solaris.setaoffice.com
Node Type : Sun SPARC (HTTPS)
Severity : normal
OM Server Time: 2016-09-10 08:03:10
Message : UXMON: SSHD Daemon is not running or not doing it properly, please check
Msg Group : OS
Application : sshd_mon
Object : sshd
Event Type :
not_found

Instance Name :
not_found

Instruction : It has been detected an SSH installation but the SSHD is not running
Please check SSH status, because it might happen also there are still some ssh spawned processes running but the father has died.

Note that if the SSH is not available this might prevent users log in the server and even impact some applications.

HPOM is complaining that ssh is not running but obviously is running because you’re connected to the server using ssh

root@solaris:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check sshdmon
Fri Sep 23 11:45:46 2016 : INFO : UXMONsshdmon is running now, pid=2250
Fri Sep 23 11:45:46 2016 : SSHDMON: SSHD – Not running
mv: /dev/null and /dev/null are identical
Fri Sep 23 11:45:46 2016 : INFO : UXMONsshdmon end, pid=2250

Check directory /var/run

root@solaris:/var/run # ls -la
total 16
drwxr-xr-x 4 root other 5 Sep 23 11:50 .
drwxr-xr-x 44 root sys 50 Aug 16 11:04 ..
-rw——- 1 root root 6 Jul 7 11:24 ds_agent.pid
drwxr-xr-x 13 root root 13 Aug 10 15:33 install_engine
drwx–x–x 2 root sys 2 Jul 6 14:27 sudo

It should have many files in /var/run

root@solaris:/var/run # ls -l
total 272
-rw——- 1 root root 0 Sep 10 21:27 AdDrEm.lck
drwxr-xr-x 3 root sys 183 Sep 10 21:43 cacao
-rw-rw-rw- 1 root bin 14 Sep 23 09:20 cdrom_rcm.conf
drwxr-xr-x 2 daemon daemon 183 Sep 23 12:18 daemon
-rw-r—– 1 root root 6 Sep 23 10:41 did_reloader.lock
-rw——- 1 root root 5 Sep 10 21:27 ds_agent.pid
Drw-r—– 1 root root 0 Sep 10 21:28 event_listener_proxy_door
Drw-r–r– 1 root root 0 Sep 10 21:40 fed_doorglobal
Drw-r–r– 1 root root 0 Sep 10 21:27 hotplugd_door
Drw-r–r– 1 root root 0 Sep 10 21:28 ifconfig_proxy_doorglobal
-rw——- 1 root root 0 Sep 10 21:26 ipsecconf.lock
Dr–r–r– 1 daemon daemon 0 Sep 10 21:26 kcfd_door
-rw——- 1 root root 0 Sep 14 09:07 lockf_raidctl
Dr–r–r– 1 root root 0 Sep 10 21:26 name_service_door
-rw-r–r– 1 root root 8 Sep 10 21:40 nfs4_domain
drwxr-xr-x 2 root root 179 Sep 10 21:40 pcmcia
Dr–r–r– 1 root root 0 Sep 10 21:26 picld_door
Drw-r–r– 1 root root 0 Sep 10 21:30 pmfd_doorglobal
-rw-r–r– 1 root sys 58 Sep 10 21:30 psn
Dr——– 1 root root 0 Sep 10 21:26 rcm_daemon_door
-rw-r–r– 1 root root 0 Sep 10 21:26 rcm_daemon_lock
-rw——- 1 root root 1068 Sep 10 21:26 rcm_daemon_state
Drw-r–r– 1 root root 0 Sep 10 21:40 rgmd_receptionist_doorglobal
drwxrwxrwt 2 root root 186 Sep 10 21:27 rpc_door
drwx—— 2 root root 182 Sep 10 21:27 smc898
-rw-r–r– 1 root root 5 Sep 10 21:27 sshd.pid
drwx–x–x 3 root sys 176 Sep 10 21:31 sudo
drwxr-xr-x 3 root root 191 Sep 10 21:26 sysevent_channels
Drw-r–r– 1 root root 0 Sep 10 21:30 sysevent_proxy_doorglobal
-rw-r–r– 1 root root 5 Sep 10 21:27 syslog.pid
Drw-r–r– 1 root root 0 Sep 10 21:27 syslog_door
-rw-r–r– 1 root root 8192 Sep 10 21:26 tzsync
drwx—— 2 root root 2625 Sep 23 10:26 zones
Drw-r–r– 1 root root 0 Sep 10 21:30 zoneup_doorglobal

Fixing the issue for the ticket. Check ssh processes for root user

root@solaris:/var/run # ps -ef | grep ssh | grep root
root 8047 1 0 Sep 21 ? 0:00 /usr/lib/ssh/sshd
root 17380 13924 0 00:07:02 ? 0:00 /usr/lib/ssh/sshd
root 5570 13878 0 08:08:40 ? 0:00 /usr/lib/ssh/sshd
root 13877 1 0 Sep 21 ? 0:00 /usr/lib/ssh/sshd
root 1003 13878 0 09:17:01 ? 0:00 /usr/lib/ssh/sshd
root 13903 1 0 Sep 21 ? 0:00 /usr/lib/ssh/sshd
root 60966 13918 0 00:03:07 ? 0:00 /usr/lib/ssh/sshd
root 48654 13878 0 10:13:22 ? 0:00 /usr/lib/ssh/sshd
root 13918 1 0 Sep 21 ? 0:00 /usr/lib/ssh/sshd
root 17389 13924 0 00:07:02 ? 0:00 /usr/lib/ssh/sshd
root 39554 13878 0 09:21:51 ? 0:00 /usr/lib/ssh/sshd
root 64681 1 0 11:25:02 ? 0:00 /usr/lib/ssh/sshd
root 11912 13878 0 09:29:14 ? 0:00 /usr/lib/ssh/sshd
root 56172 13878 0 11:54:55 ? 0:00 /usr/lib/ssh/sshd
root 17386 13924 0 00:07:02 ? 0:00 /usr/lib/ssh/sshd
root 34708 13878 0 08:51:07 ? 0:00 /usr/lib/ssh/sshd
root 60201 13878 0 09:27:36 ? 0:00 /usr/lib/ssh/sshd
root 55272 1 0 11:54:33 ? 0:00 /usr/lib/ssh/sshd
root 5850 13878 0 08:08:47 ? 0:00 /usr/lib/ssh/sshd
root 9865 44290 0 11:56:17 pts/4 0:00 grep ssh
root 13924 1 0 Sep 21 ? 0:00 /usr/lib/ssh/sshd
root 13878 1 0 Sep 21 ? 0:01 /usr/lib/ssh/sshd

Creating file /var/run/sshd.pid with sshd PID

echo 8047 > /var/run/sshd.pid

root@solaris:/var/run # ls -l sshd.pid
-rw-r–r– 1 root root 5 Sep 10 21:27 sshd.pid

sshdmon does not complain anymore

root@solaris:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check sshdmon
Fri Sep 23 11:58:15 2016 : INFO : UXMONsshdmon is running now, pid=18095
mv: /dev/null and /dev/null are identical
Fri Sep 23 11:58:15 2016 : INFO : UXMONsshdmon end, pid=18095

Solaris Volume Manager – Delete replicas of the metadevice state database

In this Solaris server, one of the disks needs replacement

root@solaris # echo | format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t2d0 <drive not available>
/pci@1f,0/pci@1,1/scsi@2/sd@2,0
1. c0t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1f,0/pci@1,1/scsi@2/sd@3,0
Specify disk (enter its number): Specify disk (enter its number):

Checking replicas of the metadevice state database

root@solaris # metadb
flags first blk block count
M p 16 unknown /dev/dsk/c0t2d0s4
M p 8208 unknown /dev/dsk/c0t2d0s4
M p 16400 unknown /dev/dsk/c0t2d0s4
M p 16 unknown /dev/dsk/c0t2d0s5
M p 8208 unknown /dev/dsk/c0t2d0s5
M p 16400 unknown /dev/dsk/c0t2d0s5
a m p lu 16 8192 /dev/dsk/c0t3d0s4
a p l 8208 8192 /dev/dsk/c0t3d0s4
a p l 16400 8192 /dev/dsk/c0t3d0s4
a p l 16 8192 /dev/dsk/c0t3d0s5
a p l 8208 8192 /dev/dsk/c0t3d0s5
a p l 16400 8192 /dev/dsk/c0t3d0s5

Deleting metadevice state database on the slices of the bad disk. First slice 4

root@solaris # metadb -d /dev/dsk/c0t2d0s4
metadb: solaris: Bad address

root@solaris # metadb
flags first blk block count
M p 16 unknown /dev/dsk/c0t2d0s5
M p 8208 unknown /dev/dsk/c0t2d0s5
M p 16400 unknown /dev/dsk/c0t2d0s5
a m p lu 16 8192 /dev/dsk/c0t3d0s4
a p l 8208 8192 /dev/dsk/c0t3d0s4
a p l 16400 8192 /dev/dsk/c0t3d0s4
a p l 16 8192 /dev/dsk/c0t3d0s5
a p l 8208 8192 /dev/dsk/c0t3d0s5
a p l 16400 8192 /dev/dsk/c0t3d0s5

And then in slice 5

root@solaris # metadb -d /dev/dsk/c0t2d0s5
metadb: solaris: Bad address

root@solaris # metadb
flags first blk block count
a m p lu 16 8192 /dev/dsk/c0t3d0s4
a p l 8208 8192 /dev/dsk/c0t3d0s4
a p l 16400 8192 /dev/dsk/c0t3d0s4
a p l 16 8192 /dev/dsk/c0t3d0s5
a p l 8208 8192 /dev/dsk/c0t3d0s5
a p l 16400 8192 /dev/dsk/c0t3d0s5

Suppressing EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full! error in HPOM

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : minor
OM Server Time: 2016-06-18 04:38:03
Message : EXT4-fs: warning (device dm-134): ext4_dx_add_entry: Directory index full!
Msg Group : OS
Application : dmsg_mon
Object : EXT4
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Check if the file /var/opt/OV/conf/OpC/dmsg_mon.cfg
This is the error message showing on dmesg

root@linux:~ # dmesg
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!

I’m going to suppress in HPOM. First check if the file exists and if it doesn’t make a copy from instrumentation directory

root@linux:~ # ls -l /var/opt/OV/conf/OpC/dmsg_mon.cfg
ls: cannot access /var/opt/OV/conf/OpC/dmsg_mon.cfg: No such file or directory

root@linux:~ # cp /var/opt/OV/bin/instrumentation/dmsg_mon.cfg /var/opt/OV/conf/OpC/

Edit the file and add the string complained by dmsg_mon

root@linux:~ # vi /var/opt/OV/conf/OpC/dmsg_mon.cfg
###############################################################################
#@(#) $Id: dmsg_mon.cfg 2132 2014-08-22 06:47:32Z zhaofeif $
#@(#) $Rev: 2132 $
#@(#) $Author: zhaofeif $
#@(#) $Date: 2014-08-22 14:47:32 +0800 (Fri, 22 Aug 2014) $
#@(#) $LastChangedBy: zhaofeif $
###############################################################################

###############################################################################
#
# File: dmsg_mon.
# [disable = yes|no]
# [interval = ]
# disable
#===============
# If set disable to YES (or yes), this module won’t run anytime
#
# interval
#===============
# If the module will allow to run after the interval minutes

# Description: strings listed here don’t generate an ITO message for dmesg
# Syntax: just list the strings, one line for each
# !!! all dmesg lines matching one of the listed strings
# are taken out of monitoring !!!
#
# Example:
#
# hardware path
#
# If the string “hardware path” is listed, all dmesg lines matching (containing)
# the string “hardware path” are ignored for monitoring purposes.
# Still, the dmesg history contains these lines, but no message is generated.
#
###############################################################################

###############################################################################
# End of dmesg_mon.cfg
###############################################################################

EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!

Run UXMONbroker. The module related to the dmsg_mon.cfg template is dmesg

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -h

GD UXMON monitoring package
Broker utility

usage: /var/opt/OV/bin/instrumentation/UXMONbroker [-h | –help] [-x ] [ -d ] [-l ] [-c ] [-f]
[ –col -p param ]
[ -t ] [ -p ] [-b ]
[ –os ]
[ -v ]

-h : this (help) message
–help : this (help) message

-x : triggers the execution of the module passed as parameter
-d : Allows execution activating debug
-check : Same as -x but output is redirected standard output, no logfile used
-l : output the logfile used by the module passed as parameter
-c : output the preferred config file used by the module

-t : output the TEMPORAL folder to be used if needed
-b : output the folder where the commands or instrumentation are located
-perl : output the perl runtime to be used

–col : Execute the collecting information of module
-p : Parameter passed to the recollection
–os : Show the OS name

-v : Version of UXMON package
-f : force the execution of the module bypass interval setting
supported modules are:
actmon, sshdmon, uxmon, nfsmon, selfcheck, swapmon, evm, mpmon, mdmon, cronmon, bondmon, rcmon, volmon, scmon, loopmon, dmesg, advfsmon, ntpmon, hwmon, bootmon, nicmon, perfmon, psmon, lpmon, vcmon, ktsmon, sgmon, dfmon.
This is the interface to the OVO templates. Templates will call this command
to get executed the different modules available, or retrieve configuration
information about the UXMON and the platform

Run UXMONbroker with dmesg

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -d dmesg
>>Debug mode activated
>>Opened the logfile: /var/opt/OV/log/OpC/dmsg_mon.log
>>logfile: /var/opt/OV/log/OpC/dmsg_mon.log
>>history file: /var/opt/OV/log/OpC/dmsg_mon.hist[1-5]
>>config line: EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
>>Module interval setting: 0
>>process info: 108389 /opt/OV/nonOV/perl/a/bin/perl -I/var/opt/OV/bin/instrumentation /var/opt/OV/bin/instrumentation/UXMONdmsg -d -c /var/opt/OV/conf/OpC/dmsg_mon.cfg -l /var/opt/OV/log/OpC/dmsg_mon.log
>>configuration file: /var/opt/OV/conf/OpC/dmsg_mon.cfg
>>lock file: /var/opt/OV/tmp/OpC/UXMONdmsg_dmsg_mon.lock
>>get lock to read /var/opt/OV/tmp/OpC/UXMONdmsg_dmsg_mon.lock
>>run the process now
>>Logged this info…:Wed Jun 22 14:50:56 2016 : INFO : UXMONdmsg is running now, pid=108389
>>There are the following lines in actual dmesg buffer program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!

>>This is the old dmesg buffer program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
program bma is using a deprecated SCSI ioctl, please convert it to SG_IO
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!
EXT4-fs warning (device dm-134): ext4_dx_add_entry: Directory index full!

>>Difference between old and new dmesg buffers
>>The following is assigned to dmesg for processing
>>ReleaseLock: 108389 1466617856 0 stopped 108389 /opt/OV/nonOV/perl/a/bin/perl -I/var/opt/OV/bin/instrumentation /var/opt/OV/bin/instrumentation/UXMONdmsg -d -c /var/opt/OV/conf/OpC/dmsg_mon.cfg -l /var/opt/OV/log/OpC/dmsg_mon.log
>>Logged this info…:Wed Jun 22 14:50:56 2016 : INFO : UXMONdmsg end, pid=108389

The ‘directory index full’ error will be seen if there are lots of files/directories in the filesystem so that the tree reaches its indexing limits and cannot keep track further.

The directory index is an additional tree structure which helps directory lookups, improving performance for huge directories.
Source: https://access.redhat.com/solutions/29894

HPE Fibreutils

Fibreutils is a set of utility scripts for Linux that make certain operations, such as information gathering, easier. The following sections describe commands contained in the Fibreutils
package.

adapter_info

The adapter_info command lists information about the Fibre Channel adapters.

–I Lists all FC adapters in this system.
–L Lists all LUNs for each adapter.
–t Lists all targets seen by each adapter.
-m Lists each adapter’s model.
–v Verbose
–h Prints the help menu.
–d Prints LUN and verbose information for a specific device. The specific device should be a SCSI host number such as 0 or 6.

lssd

The lssd command lists all bound /dev/sd* devices.

–c Prints cached data instead of scanning.
–h Prints help message.
–l Prints inquiry page 0 x 83 UUID of devices.
–w Prints WWNN of devices.

lssg

The lssg command lists all bound /dev/sg* devices.

–c Prints cached data instead of scanning.
–h Prints help message.
–l Prints inquiry page 0 x 83 UUID of devices.
–w Print WWNN of devices.

hp_rescan

The hp_rescan command rescans LUNs on Hewlett Packard Enterprise supported Fibre Channel
adapters.

–a Rescans all adapters.
–h Prints help message
–i Rescans a specific adapter instance. The specific device should be a SCSI host number such as 0 or 6.
–l Lists all FC adapters.
–n Do not perform scsi remove-single-device when executing probe-luns.

Locating adapter_info

root@linux:~ # which adapter_info
/usr/bin/adapter_info

It is provided by the package fibreutils

root@linux:~ # rpm -qf /usr/bin/adapter_info
fibreutils-3.2-6

root@linux:~ # rpm -qi fibreutils-3.2-6
Name : fibreutils Relocations: (not relocatable)
Version : 3.2 Vendor: Hewlett-Packard Company
Release : 6 Build Date: Tue 18 Dec 2012 02:38:23 AM BRST
Install Date: Wed 26 Mar 2014 10:42:41 AM BRT Build Host: localhost.localdomain
Group : Applications/System Source RPM: fibreutils-3.2-6.src.rpm
Size : 77657 License: Proprietary
Signature : RSA/SHA1, Tue 18 Feb 2014 04:44:32 PM BRT, Key ID b070680a5ce2d476
Packager : Hewlett-Packard Company
URL : http://www.hp.com
Summary : Provides complimentary programs and scripts for HP supported fibre channel HBAs
Description :
This RPM has the following components:

* Miscellaneous scripts and programs to compliment HP supported FC drivers:

lssd
lssg
adapter_info
hp_rescan
scsi_info

Help message for adapter_info

root@linux:~ # /usr/bin/adapter_info -h
NAME

adapter_info

DESCRIPTION

Prints information about Fibre Channel HBAs/CNAs.

OPTIONS

-d, –device – Prints all information for a specific SCSI host adapter
-h, –help – Prints this help message
-i, –versioninfo – Prints driver version information for all HBAs
-l, –luns – Prints the device information for all HBAs
-m, –model – Prints the HBA model for all HBAs
-p, –pciids – Prints the PCI IDs for all HBAs
-r, –remoteports – Prints the attached remote ports for all HBAs
-v, –verbose – Prints all information except device and LUN information

Checking HBA card WWNN WWPN and state

root@linux:~ # /usr/bin/adapter_info
/sys/class/scsi_host/host3: wwnn=0x2000001b32001275 wwpn=0x2100001b32001275 state=Link Up – F_Port
/sys/class/scsi_host/host4: wwnn=0x2000001b3200c270 wwpn=0x2100001b3200c270 state=Link Up – F_Port
/sys/class/scsi_host/host1: wwnn=0x20000000c97e3572 wwpn=0x10000000c97e3572 state=Online
/sys/class/scsi_host/host2: wwnn=0x20000000c97e3573 wwpn=0x10000000c97e3573 state=Linkdown

Source: https://h20565.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=461311&docId=emr_na-c01487184&docLocale=en_US

%d bloggers like this: