Advertisements

Category Archives: Linux

Creating a new logical drive with two physical drives mirrored on HPE Proliant

Checking disks

root@linux:~ # hpacucli ctrl all show config

Smart Array P410i in Slot 0 (Embedded) (sn: 5001438013682F50)

array A (SAS, Unused Space: 0 MB)

logicaldrive 1 (136.7 GB, RAID 1, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 146 GB, OK)

array B (SAS, Unused Space: 0 MB)

logicaldrive 2 (136.7 GB, RAID 1, OK)

physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 146 GB, OK)

array C (SAS, Unused Space: 0 MB)

logicaldrive 3 (136.7 GB, RAID 1, OK)

physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)

SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 5001438013682F5F)

Listing logical drives

root@linux:~ # hpacucli ctrl slot=0 logicaldrive all show status

logicaldrive 1 (136.7 GB, RAID 1): OK
logicaldrive 2 (136.7 GB, RAID 1): OK
logicaldrive 3 (136.7 GB, RAID 1): OK

Showing physical drives

root@linux:~ # hpacucli ctrl slot=0 pd all show

Smart Array P410i in Slot 0 (Embedded)

array A

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 146 GB, OK)

array B

physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 146 GB, OK)

array C

physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)

Two disks were inserted

Checking status again

root@linux:~ # hpacucli ctrl slot=0 pd all show

Smart Array P410i in Slot 0 (Embedded)

array A

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 146 GB, OK)

array B

physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 146 GB, OK)

array C

physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)

unassigned

physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)

Creating a new logical drive with mirror

root@linux:~ # hpacucli ctrl slot=0 create type=ld drives=2I:1:7,2I:1:8 raid=1

root@linux:~ # hpacucli ctrl all show config

Smart Array P410i in Slot 0 (Embedded) (sn: 5001438013682F50)

array A (SAS, Unused Space: 0 MB)

logicaldrive 1 (136.7 GB, RAID 1, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 146 GB, OK)

array B (SAS, Unused Space: 0 MB)

logicaldrive 2 (136.7 GB, RAID 1, OK)

physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 146 GB, OK)

array C (SAS, Unused Space: 0 MB)

logicaldrive 3 (136.7 GB, RAID 1, OK)

physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SAS, 146 GB, OK)
physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS, 146 GB, OK)

array D (SAS, Unused Space: 0 MB)

logicaldrive 4 (136.7 GB, RAID 1, OK)

physicaldrive 2I:1:7 (port 2I:box 1:bay 7, SAS, 146 GB, OK)
physicaldrive 2I:1:8 (port 2I:box 1:bay 8, SAS, 146 GB, OK)

SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 5001438013682F5F)

Advertisements

Red Hat Enteprise Linux/CENTOS 6 with Mellanox network interface stopped working after reboot: mlx4_en.ko needs unknown symbol

In a upgrade, after installing the kernel-2.6.32-696.20.1.el6.x86_64 on a Red Hat 6 it showed the following message

mlx4_en.ko needs unknown symbol mlx4_SET_PORT_BEACON

The network interface were not working after the reboot

4 network interfaces that stopped working – eth0, eth1, eth2, eth3

They use the Mellanox driver

root@linux:~ # ethtool -i eth0
driver: mlx4_en
version: 4.1-1.0.2 (27 Jun 2017)
firmware-version: 2.35.5100
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

root@linux:~ # modinfo mlx4_en
filename: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-en/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko
version: 2.3-2.0.1 (Dec 7 2014)
license: Dual BSD/GPL
description: Mellanox ConnectX HCA Ethernet driver
author: Liran Liss, Yevgeny Petrilin
srcversion: 86DD1A5FE5CF17268AFBB0D
depends: mlx4_core,compat,ptp
vermagic: 2.6.32-504.el6.x86_64 SMP mod_unload modversions
parm: inline_thold:threshold for using inline data (uint)
parm: udp_rss:Enable RSS for incoming UDP traffic (uint)
parm: num_lro:Dummy module parameter to prevent loading issues (uint)
parm: pfctx:Priority based Flow Control policy on TX[7:0]. Per priority bit mask (uint)
parm: pfcrx:Priority based Flow Control policy on RX[7:0]. Per priority bit mask (uint)

You need to install the Mellanox driver in order for the network interfaces to work

The server is a HPE Proliant DL380 Gen9. Download both files to install and copy them to the server. If necessary, boot the old working kernel

HPE Mellanox RoCE (RDMA over Converged Ethernet) Driver for Red Hat Enterprise Linux 6 Update 6 (x86_64)

HPE Mellanox RoCE (RDMA over Converged Ethernet) Driver for Red Hat Enterprise Linux 6 Update 7 (x86_64)

HPE Mellanox RoCE (RDMA over Converged Ethernet) Driver for Red Hat Enterprise Linux 6 Update 8 (x86_64)

HPE Mellanox RoCE (RDMA over Converged Ethernet) Driver for Red Hat Enterprise Linux 6 Update 9 (x86_64)

Copied both files to /tmp

kmod-mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.rhel6u9.x86_64.rpm
mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.rhel6u9.x86_64.rpm

In my case I had to remove the following packages and dependencies

# yum remove librdmacm libibverbs
# yum remove mlnx-en-utils kmod-mlnx-en

Then install both packages

root@linux:/tmp # rpm -ivh mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.rhel6u9.x86_64.rpm kmod-mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.rhel6u9.x86_64.rpm
warning: mlnx-ofa_kernel-4.1-OFED.4.1.1.0.2.1.gc22af88.rhel6u9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 26c2b797: NOKEY
Preparing… ########################################### [100%]
1:mlnx-ofa_kernel ########################################### [ 50%]
Configured /etc/security/limits.conf
2:kmod-mlnx-ofa_kernel ########################################### [100%]
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/kernel/drivers/net/mlx4/mlx4_en.ko needs unknown symbol mlx4_test_interrupts
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/kernel/drivers/net/mlx4/mlx4_en.ko needs unknown symbol mlx4_set_stats_bitmap
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko needs unknown symbol ib_query_device
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko needs unknown symbol ib_alloc_pd
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_rq_tracked
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_q_counter
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_nic_vport_qkey_viol_cntr
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_sq_tracked
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_attach_nvmf_ns
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_nic_vport_update_local_lb
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_hca_vport_context
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rl_add_rate
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_create_auto_grouped_flow_table
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_rqt
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_node_guid
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_vendor_id
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_rq
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_alloc_q_counter
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_dct
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_set_delay_drop
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_dealloc_q_counter
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_arm_dct
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_tis
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_tir
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_cmd_create_vport_lag
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_unregister_interface
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_min_inline
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_dealloc_transport_domain
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_nic_vport_system_image_guid
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_lag_get_roce_netdev
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_add_flow_rules
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_destroy_flow_table
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_get_flow_namespace
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_tir
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_put_uars_page
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_get_uars_page
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_nic_vport_node_guid
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_detach_nvmf_ns
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_nvmf_backend_ctrl
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_del_flow_rules
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rdma_netdev_alloc
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_sq
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_system_image_guid
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_vport_counter
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_nvmf_backend_ctrl
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_max_mtu
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_gid
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_nic_vport_disable_roce
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_create_lag_demux_flow_table
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_sq_tracked
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rl_remove_rate
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_mkey_cb
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_eth_proto_oper
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_rq_tracked
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_set_dc_cnak_trace
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_rqt
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_dct_query
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rdma_netdev_free
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_vl_hw_cap
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_pkey
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_ib_proto_oper
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_context
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_oper_mtu
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_register_interface
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_nic_vport_enable_roce
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_rq
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_tis
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_tis
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_link_width_oper
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_cq_moderation
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_dct
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_ib_ppcnt
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_alloc_bfreg
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_vector2eqn
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_lag_is_active
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_free_bfreg
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_sq
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_alloc_transport_domain
WARNING: /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_cmd_destroy_vport_lag
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/kernel/drivers/net/mlx4/mlx4_en.ko needs unknown symbol mlx4_test_interrupts
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/kernel/drivers/net/mlx4/mlx4_en.ko needs unknown symbol mlx4_set_stats_bitmap
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko needs unknown symbol ib_query_device
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko needs unknown symbol ib_alloc_pd
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_rq_tracked
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_q_counter
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_nic_vport_qkey_viol_cntr
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_sq_tracked
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_attach_nvmf_ns
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_nic_vport_update_local_lb
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_hca_vport_context
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rl_add_rate
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_create_auto_grouped_flow_table
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_rqt
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_node_guid
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_vendor_id
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_rq
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_alloc_q_counter
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_dct
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_set_delay_drop
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_dealloc_q_counter
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_arm_dct
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_tis
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_tir
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_cmd_create_vport_lag
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_unregister_interface
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_min_inline
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_dealloc_transport_domain
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_nic_vport_system_image_guid
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_lag_get_roce_netdev
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_add_flow_rules
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_destroy_flow_table
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_get_flow_namespace
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_tir
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_put_uars_page
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_get_uars_page
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_nic_vport_node_guid
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_detach_nvmf_ns
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_nvmf_backend_ctrl
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_del_flow_rules
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rdma_netdev_alloc
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_sq
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_system_image_guid
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_vport_counter
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_nvmf_backend_ctrl
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_max_mtu
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_gid
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_nic_vport_disable_roce
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_create_lag_demux_flow_table
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_sq_tracked
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rl_remove_rate
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_mkey_cb
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_eth_proto_oper
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_create_rq_tracked
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_set_dc_cnak_trace
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_rqt
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_dct_query
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_rdma_netdev_free
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_vl_hw_cap
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_pkey
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_ib_proto_oper
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_hca_vport_context
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_oper_mtu
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_register_interface
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_nic_vport_enable_roce
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_rq
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_tis
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_tis
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_query_port_link_width_oper
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_modify_cq_moderation
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_destroy_dct
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_ib_ppcnt
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_alloc_bfreg
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_vector2eqn
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_lag_is_active
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_free_bfreg
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_query_sq
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_core_alloc_transport_domain
WARNING: /lib/modules/2.6.32-573.22.1.el6.x86_64/weak-updates/mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko needs unknown symbol mlx5_cmd_destroy_vport_lag

Reboot the server

root@linux:~ # shutdown -r now

Broadcast message from root (pts/0) (Wed Feb 21 16:13:26 2018):

The system is going down for reboot NOW!

HPOM – Can’t open perl script “/var/opt/OV/bin/instrumentation/UXMONperfmon”: No such file or directory

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -help

GD UXMON monitoring package
Broker utility

usage: /var/opt/OV/bin/instrumentation/UXMONbroker [-h | –help] [-x ] [ -d ] [-l ] [-c ] [-f]
[ –col -p param ]
[ -t ] [ -p ] [-b ]
[ –os ]
[ -v ]

-h : this (help) message
–help : this (help) message

-x : triggers the execution of the module passed as parameter
-d : Allows execution activating debug
-check : Same as -x but output is redirected standard output, no logfile used
-l : output the logfile used by the module passed as parameter
-c : output the preferred config file used by the module

-t : output the TEMPORAL folder to be used if needed
-b : output the folder where the commands or instrumentation are located
-perl : output the perl runtime to be used

–col : Execute the collecting information of module
-p : Parameter passed to the recollection
–os : Show the OS name

-v : Version of UXMON package
-f : force the execution of the module bypass interval setting
supported modules are:
actmon, sshdmon, tdfmon, uxmon, nfsmon, selfcheck, swapmon, evm, mpmon, mdmon, cronmon, bondmon, rcmon, volmon, scmon, loopmon, dmesg, advfsmon, ntpmon, hwmon, bootmon, nicmon, perfmon, psmon, lpmon, vcmon, ktsmon, sgmon, dfmon.
This is the interface to the OVO templates. Templates will call this command
to get executed the different modules available, or retrieve configuration
information about the UXMON and the platform

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check perfmon
Can’t open perl script “/var/opt/OV/bin/instrumentation/UXMONperfmon”: No such file or directory

root@linux:~ # ls -l /var/opt/OV/log/OpC/perf_mon.log
-rw-rw-r– 1 root bin 0 Feb 1 23:00 /var/opt/OV/log/OpC/perf_mon.log

Linux LVM: Device /dev/mapper/ECP_data_disk_001 has size of 1048576000 sectors which is smaller than corresponding PV size of 1174372352 sectors. Was device resized?

Running LVM command and it shows an unrelated disk that is complaining it has a unexpected size.

This problem was happening because it is a cluster node and LUN was resized.

On the node that showed the message, I didn’t run any commands to resize it

root@linux02:~ # pvs | grep PP0_data_disk_001
Device /dev/mapper/ECP_data_disk_001 has size of 1048576000 sectors which is smaller than corresponding PV size of 1174372352 sectors. Was device resized?

Run the commands to resize the LUN on the node

root@linux02:~ # multipath -ll ECP_data_disk_001 | grep ready | awk ‘{print “echo 1 > /sys/block/”$3″/device/rescan”‘} | bash
root@linux02:~ # multipathd -k’resize map ECP_data_disk_001′
ok
root@linux02:~ # pvresize /dev/mapper/ECP_data_disk_001
Physical volume “/dev/mapper/ECP_data_disk_001” changed
1 physical volume(s) resized / 0 physical volume(s) not resized

EXT4-fs: : Can’t allocate: Allocation context details

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : minor
OM Server Time: 2017-12-29 15:04:09
Message : EXT4-fs: : Can’t allocate: Allocation context details:
Msg Group : OS
Application : dmsg_mon
Object : EXT4
Event Type :
not_found

Instance Name :
not_found

Instruction : No
EventDataSource :

This problem happens because there is a filesystem nearly full

root@linux:~ # df -hP /osprov
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg01-ospvol 405G 379G 4.4G 99% /osprov

root@linux:~ # vgs vg01
VG #PV #LV #SN Attr VSize VFree
vg01 1 4 0 wz–n- 599.88G 11.25G

Checking the logs, I discovered that the device being affected is a snapshot from a filesystem

root@linux:/var/log # zgrep “Can’t allocate: Allocation context details” messages* | grep -v CMD
messages.1.gz:Dec 8 20:00:16 linux kernel: EXT4-fs: Can’t allocate: Allocation context details:
messages.1.gz:Dec 29 20:00:17 linux kernel: EXT4-fs: Can’t allocate: Allocation context details:
messages.2.gz:Nov 10 20:00:17 linux kernel: EXT4-fs: Can’t allocate: Allocation context details:
messages.2.gz:Nov 24 20:00:17 linux kernel: EXT4-fs: Can’t allocate: Allocation context details:

Getting ” EXT4-fs: Can’t allocate: Allocation context details” errors in logs while using ext4 with Red Hat Enterprise Linux 5.6 and higher

I’ve added message to file /var/opt/OV/conf/OpC/dmsg_mon.cfg to ignore this alarm

root@linux:/var/opt/OV/conf/OpC # vi dmsg_mon.cfg
EXT4-fs: Can’t allocate: Allocation context details

How to uncompress a .xz file in Red Hat Enterprise Linux

You have generated a sosreport and now want to take a look at the files. To extract the files you need to have installed xz

root@linux:~ # rpm -qi xz
Name : xz
Version : 5.2.2
Release : 1.el7
Architecture: x86_64
Install Date: Thu 05 Jan 2017 12:36:49 PM EST
Group : Applications/File
Size : 798066
License : LGPLv2+
Signature : RSA/SHA256, Thu 05 May 2016 10:16:16 AM EDT, Key ID 199e2f91fd431d51
Source RPM : xz-5.2.2-1.el7.src.rpm
Build Date : Fri 04 Mar 2016 07:57:32 AM EST
Build Host : x86-024.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager : Red Hat, Inc.
Vendor : Red Hat, Inc.
URL : http://tukaani.org/xz/
Summary : LZMA compression utilities
Description :
XZ Utils are an attempt to make LZMA compression easy to use on free (as in
freedom) operating systems. This is achieved by providing tools and libraries
which are similar to use than the equivalents of the most popular existing
compression algorithms.

LZMA is a general purpose compression algorithm designed by Igor Pavlov as
part of 7-Zip. It provides high compression ratio while keeping the
decompression speed fast.

Then run unxz and extract the tarball with tar

root@linux:/var/tmp # unxz sosreport-linux.setaoffice.com-20171212113141.tar.xz
root@linux:/var/tmp # tar -xf sosreport-linux.setaoffice.com-20171212113141.tar

UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy

Also see:
UXMON: mpathb – Only one path detected, no path redundancy
UXMON: volumegroup – Only one path detected, no path redundancy

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2017-12-11 03:28:31
Message : UXMON: SY1_log2_disk_001 – Only one path detected, no path redundancy
Msg Group : OS
Application : mpmon
Object : mp
Event Type :
not_found

Instance Name :
not_found

Instruction : The multipathd -k”show map $device topology” command shows more details

Please check /var/opt/OV/log/OpC/mp_mon.log for more details
EventDataSource :

When running a LVM command it is showing that several PVs are duplicate.

root@linux:~ # pvs
Found duplicate PV s3zEE42awIhydJ05hfUzJulPsN8WS266: using /dev/mapper/SY1_disknew_001 not /dev/sdbm
Using duplicate PV /dev/mapper/SY1_disknew_001 from subsystem DM, replacing /dev/sdbm
Found duplicate PV zTZlnwYgW69xzUcTi0riu5euTPiWWnRs: using /dev/mapper/swap_disk_001p1 not /dev/sdbo1
Using duplicate PV /dev/mapper/swap_disk_001p1 from subsystem DM, ignoring /dev/sdbo1
Found duplicate PV hnuWyziFeXRhJ71YpLJX1cLAjOvgtl0q: using /dev/mapper/SCR_DATA_disk_001p1 not /dev/sdbn1
Using duplicate PV /dev/mapper/SCR_DATA_disk_001p1 from subsystem DM, replacing /dev/sdbn1
Found duplicate PV EeqL0LDdMohfnagni16NlRUig3eugbap: using /dev/mapper/SY1_log2_disk_001p1 not /dev/sdbp1
Using duplicate PV /dev/mapper/SY1_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbp1
Found duplicate PV sMAYQABFedTStD589d5ZcDu1ZNtXfXyh: using /dev/mapper/SY1_log1_disk_001p1 not /dev/sdbq1
Using duplicate PV /dev/mapper/SY1_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbq1
Found duplicate PV qFBDDPa7FtN7F97tpUDeKv0cBO94WBr3: using /dev/mapper/SCR_disk_002 not /dev/sdbr
Using duplicate PV /dev/mapper/SCR_disk_002 from subsystem DM, ignoring /dev/sdbr
Found duplicate PV RZvwCnQn48G2A9IifNOLZT9l9ZYdE7yu: using /dev/mapper/SY1_arch_disk_001p1 not /dev/sdbs1
Using duplicate PV /dev/mapper/SY1_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbs1
Found duplicate PV qWPh0v3E1ADowxUuuC8kP8BFJ1bwBsmL: using /dev/mapper/SY1_disk_001p1 not /dev/sdbt1
Using duplicate PV /dev/mapper/SY1_disk_001p1 from subsystem DM, ignoring /dev/sdbt1
Found duplicate PV hi5F4g9FPojaD9J7KH5vHCzRKgd4jQfJ: using /dev/mapper/MR2_log2_disk_001p1 not /dev/sdbu1
Using duplicate PV /dev/mapper/MR2_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbu1
Found duplicate PV KA1AkbCOgfHFGuKazmHAacwqBNPmEyV8: using /dev/mapper/MR2_log1_disk_001p1 not /dev/sdbv1
Using duplicate PV /dev/mapper/MR2_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbv1
Found duplicate PV dy0YaJoSrlZdTM9isfY1QGS6kWBTzs6i: using /dev/mapper/MR2_data_disk_001p1 not /dev/sdbw1
Using duplicate PV /dev/mapper/MR2_data_disk_001p1 from subsystem DM, ignoring /dev/sdbw1
Found duplicate PV khSEnsJ9CC4epw0uOAsaFwlbIKS32Qyj: using /dev/mapper/MR2_arch_disk_001p1 not /dev/sdbx1
Using duplicate PV /dev/mapper/MR2_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbx1
Found duplicate PV PmTTDGWvEvDM8AMUHvqcoRJeA9g4r78D: using /dev/mapper/MRC_log2_disk_001p1 not /dev/sdbi1
Using duplicate PV /dev/mapper/MRC_log2_disk_001p1 from subsystem DM, ignoring /dev/sdbi1
Found duplicate PV ffv1TgDxRIEIyhRoEh46fMGeFuJ1tVtv: using /dev/mapper/MR2_disk_001p1 not /dev/sdby1
Using duplicate PV /dev/mapper/MR2_disk_001p1 from subsystem DM, ignoring /dev/sdby1
Found duplicate PV tCeMdOB1dFuJomhgB061M7MeZNEwptk3: using /dev/mapper/MRC_log1_disk_001p1 not /dev/sdbj1
Using duplicate PV /dev/mapper/MRC_log1_disk_001p1 from subsystem DM, ignoring /dev/sdbj1
Found duplicate PV inuuS3f39VjSuh6q7r5pEZSjzRgRhi5e: using /dev/mapper/sap_disk_001p1 not /dev/sdbz1
Using duplicate PV /dev/mapper/sap_disk_001p1 from subsystem DM, ignoring /dev/sdbz1
Found duplicate PV Ea4CVYCMjYc3eE129uYlUlZKdzhmYNpQ: using /dev/mapper/MRC_data_disk_001p1 not /dev/sdbk1
Using duplicate PV /dev/mapper/MRC_data_disk_001p1 from subsystem DM, ignoring /dev/sdbk1
Found duplicate PV Lx5viH3geNo0zSO07sHcbNlJ9nVmcGPA: using /dev/mapper/MRC_arch_disk_001p1 not /dev/sdbl1
Using duplicate PV /dev/mapper/MRC_arch_disk_001p1 from subsystem DM, ignoring /dev/sdbl1
PV VG Fmt Attr PSize PFree
/dev/mapper/MR2_arch_disk_001p1 vgMR2oraarch lvm2 a–u 19.98g 0
/dev/mapper/MR2_data_disk_001p1 vgMR2data lvm2 a–u 299.98g 99.98g
/dev/mapper/MR2_disk_001p1 vgMR2 lvm2 a–u 99.98g 20.87g
/dev/mapper/MR2_log1_disk_001p1 vgMR2log1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MR2_log2_disk_001p1 vgMR2log2 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_arch_disk_001p1 vgMRCoraarch lvm2 a–u 19.98g 0
/dev/mapper/MRC_data_disk_001p1 vgMRCdata lvm2 a–u 299.98g 39.98g
/dev/mapper/MRC_disk_002 vgMRC lvm2 a–u 149.98g 31.88g
/dev/mapper/MRC_log1_disk_001p1 vgMRClog1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_log2_disk_001p1 vgMRClog2 lvm2 a–u 9.98g 7.98g
/dev/mapper/SCR_ARCH_disk_001p1 vgSCRarch lvm2 a–u 29.96g 4.57g
/dev/mapper/SCR_DATA_disk_001p1 vgSCRdata lvm2 a–u 109.96g 8.40g
/dev/mapper/SCR_LOG1_disk_001p1 vgSCRlog1 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_LOG2_disk_001p1 vgSCRlog2 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_disk_001p1 vgSCR lvm2 a–u 49.96g 3.96g
/dev/mapper/SCR_disk_002 vgSCR lvm2 a–u 49.98g 34.98g
/dev/mapper/SY1_arch_disk_001p1 vgSY1oraarch lvm2 a–u 149.98g 0
/dev/mapper/SY1_data_disk_002 vgSY1data lvm2 a–u 2.64t 0
/dev/mapper/SY1_disk_001p1 vgSY1 lvm2 a–u 49.98g 10.52g
/dev/mapper/SY1_disknew_001 vgSY1 lvm2 a–u 59.98g 44.98g
/dev/mapper/SY1_interf_disk_001 vgSY1interface lvm2 a–u 49.98g 1008.00m
/dev/mapper/SY1_log1_disk_001p1 vgSY1log1 lvm2 a–u 19.98g 9.98g
/dev/mapper/SY1_log2_disk_001p1 vgSY1log2 lvm2 a–u 19.98g 9.98g
/dev/mapper/sap_disk_001p1 vgSAPlocal lvm2 a–u 49.98g 4.79g
/dev/mapper/swap_disk_001p1 vgswap lvm2 a–u 223.98g 0
/dev/sda2 vgroot lvm2 a–u 278.86g 198.27g

Removing disk paths that are duplicate

echo 1 > /sys/block/sdbm/device/delete
echo 1 > /sys/block/sdbo/device/delete
echo 1 > /sys/block/sdbn/device/delete
echo 1 > /sys/block/sdbp/device/delete
echo 1 > /sys/block/sdbq/device/delete
echo 1 > /sys/block/sdbr/device/delete
echo 1 > /sys/block/sdbs/device/delete
echo 1 > /sys/block/sdbt/device/delete
echo 1 > /sys/block/sdbu/device/delete
echo 1 > /sys/block/sdbv/device/delete
echo 1 > /sys/block/sdbw/device/delete
echo 1 > /sys/block/sdbx/device/delete
echo 1 > /sys/block/sdbi/device/delete
echo 1 > /sys/block/sdby/device/delete
echo 1 > /sys/block/sdbj/device/delete
echo 1 > /sys/block/sdbz/device/delete

Checking status

root@linux:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/MR2_arch_disk_001p1 vgMR2oraarch lvm2 a–u 19.98g 0
/dev/mapper/MR2_data_disk_001p1 vgMR2data lvm2 a–u 299.98g 99.98g
/dev/mapper/MR2_disk_001p1 vgMR2 lvm2 a–u 99.98g 20.87g
/dev/mapper/MR2_log1_disk_001p1 vgMR2log1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MR2_log2_disk_001p1 vgMR2log2 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_arch_disk_001p1 vgMRCoraarch lvm2 a–u 19.98g 0
/dev/mapper/MRC_data_disk_001p1 vgMRCdata lvm2 a–u 299.98g 39.98g
/dev/mapper/MRC_disk_002 vgMRC lvm2 a–u 149.98g 31.88g
/dev/mapper/MRC_log1_disk_001p1 vgMRClog1 lvm2 a–u 9.98g 7.98g
/dev/mapper/MRC_log2_disk_001p1 vgMRClog2 lvm2 a–u 9.98g 7.98g
/dev/mapper/SCR_ARCH_disk_001p1 vgSCRarch lvm2 a–u 29.96g 4.57g
/dev/mapper/SCR_DATA_disk_001p1 vgSCRdata lvm2 a–u 109.96g 8.40g
/dev/mapper/SCR_LOG1_disk_001p1 vgSCRlog1 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_LOG2_disk_001p1 vgSCRlog2 lvm2 a–u 4.96g 984.00m
/dev/mapper/SCR_disk_001p1 vgSCR lvm2 a–u 49.96g 3.96g
/dev/mapper/SCR_disk_002 vgSCR lvm2 a–u 49.98g 34.98g
/dev/mapper/SY1_arch_disk_001p1 vgSY1oraarch lvm2 a–u 149.98g 0
/dev/mapper/SY1_data_disk_002 vgSY1data lvm2 a–u 2.64t 0
/dev/mapper/SY1_disk_001p1 vgSY1 lvm2 a–u 49.98g 10.52g
/dev/mapper/SY1_disknew_001 vgSY1 lvm2 a–u 59.98g 44.98g
/dev/mapper/SY1_interf_disk_001 vgSY1interface lvm2 a–u 49.98g 1008.00m
/dev/mapper/SY1_log1_disk_001p1 vgSY1log1 lvm2 a–u 19.98g 9.98g
/dev/mapper/SY1_log2_disk_001p1 vgSY1log2 lvm2 a–u 19.98g 9.98g
/dev/mapper/sap_disk_001p1 vgSAPlocal lvm2 a–u 49.98g 4.79g
/dev/mapper/swap_disk_001p1 vgswap lvm2 a–u 223.98g 0
/dev/sda2 vgroot lvm2 a–u 278.86g 198.27g

Checking multipath devices. Some devices have only one path

root@linux:~ # multipath -ll
SCR_disk_001 (350002acb5a22374a) dm-26 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:21 sdy 65:128 active ready running
`- 1:0:0:21 sdcb 68:240 active ready running
SY1_log2_disk_001 (350002ac17cd5374a) dm-6 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:6 sdi 8:128 active ready running
MRC_arch_disk_001 (350002ac15504374a) dm-3 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:3 sdf 8:80 active ready running
MR2_data_disk_001 (350002ac17cd8374a) dm-14 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:13 sdp 8:240 active ready running
MR2_disk_001 (350002ac17cd6374a) dm-18 3PARdata,VV
size=100G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:15 sds 65:32 active ready running
SY1_log1_disk_001 (350002ac17cd4374a) dm-7 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:7 sdj 8:144 active ready running
MR2_log2_disk_001 (350002ac17ce7374a) dm-11 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:11 sdn 8:208 active ready running
MRC_disk_002 (350002ac2904a374a) dm-32 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:23 sdaa 65:160 active ready running
`- 1:0:0:23 sdcd 69:16 active ready running
SCR_ARCH_disk_001 (350002acb5a2c374a) dm-16 3PARdata,VV
size=30G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:17 sdu 65:64 active ready running
`- 1:0:0:17 sdca 68:224 active ready running
MR2_log1_disk_001 (350002ac17ce5374a) dm-12 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:12 sdo 8:224 active ready running
SY1_arch_disk_001 (350002ac17cca374a) dm-9 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:9 sdl 8:176 active ready running
SY1_disknew_001 (350002ac4c566374a) dm-4 3PARdata,VV
size=60G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:4 sdg 8:96 active ready running
SCR_LOG2_disk_001 (350002acb5a39374a) dm-17 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:18 sdv 65:80 active ready running
`- 1:0:0:18 sdq 65:0 active ready running
MR2_arch_disk_001 (350002ac17cd7374a) dm-15 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:14 sdr 65:16 active ready running
SCR_LOG1_disk_001 (350002acb5a38374a) dm-19 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:19 sdw 65:96 active ready running
`- 1:0:0:19 sdav 66:240 active ready running
MRC_data_disk_001 (350002ac15505374a) dm-2 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:2 sde 8:64 active ready running
sap_disk_001 (350002ac1fc92374a) dm-13 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:16 sdt 65:48 active ready running
SY1_disk_001 (350002ac17cbd374a) dm-10 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:10 sdm 8:192 active ready running
SY1_interf_disk_001 (350002ac0ce50374a) dm-33 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:24 sdab 65:176 active ready running
`- 1:0:0:24 sdce 69:32 active ready running
MRC_log2_disk_001 (350002ac1551a374a) dm-71 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:0 sdc 8:32 active ready running
SY1_data_disk_002 (350002ac23826374a) dm-31 3PARdata,VV
size=2.6T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:22 sdz 65:144 active ready running
`- 1:0:0:22 sdcc 69:0 active ready running
MRC_log1_disk_001 (350002ac15519374a) dm-72 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:1 sdd 8:48 active ready running
SCR_DATA_disk_001 (350002acb5a37374a) dm-23 3PARdata,VV
size=110G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:20 sdx 65:112 active ready running
SCR_disk_002 (350002ac19e73374a) dm-8 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:8 sdk 8:160 active ready running
swap_disk_001 (350002ac155d0374a) dm-5 3PARdata,VV
size=224G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
`- 0:0:0:5 sdh 8:112 active ready running

Recognize new LUNs and run multipath -v3

root@linux:~ # systool -av -c fc_host | grep “Class Device =” | awk -F’=’ {‘print $2’} | awk -F'”‘ {‘print “echo \”- – -\” > /sys/class/scsi_host/”$2″/scan”‘} | sh
root@linux:~ # multipath -v3

Paths restored

root@linux:~ # multipath -ll
SCR_disk_001 (350002acb5a22374a) dm-26 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:21 sdy 65:128 active ready running
`- 1:0:0:21 sdcb 68:240 active ready running
SY1_log2_disk_001 (350002ac17cd5374a) dm-6 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:6 sdi 8:128 active ready running
`- 1:0:0:6 sdai 66:32 active ready running
MRC_arch_disk_001 (350002ac15504374a) dm-3 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:3 sdf 8:80 active ready running
`- 1:0:0:3 sdaf 65:240 active ready running
MR2_data_disk_001 (350002ac17cd8374a) dm-14 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:13 sdp 8:240 active ready running
`- 1:0:0:13 sdap 66:144 active ready running
MR2_disk_001 (350002ac17cd6374a) dm-18 3PARdata,VV
size=100G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:15 sds 65:32 active ready running
`- 1:0:0:15 sdar 66:176 active ready running
SY1_log1_disk_001 (350002ac17cd4374a) dm-7 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:7 sdj 8:144 active ready running
`- 1:0:0:7 sdaj 66:48 active ready running
MR2_log2_disk_001 (350002ac17ce7374a) dm-11 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:11 sdn 8:208 active ready running
`- 1:0:0:11 sdan 66:112 active ready running
MRC_disk_002 (350002ac2904a374a) dm-32 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:23 sdaa 65:160 active ready running
`- 1:0:0:23 sdcd 69:16 active ready running
SCR_ARCH_disk_001 (350002acb5a2c374a) dm-16 3PARdata,VV
size=30G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:17 sdu 65:64 active ready running
`- 1:0:0:17 sdca 68:224 active ready running
MR2_log1_disk_001 (350002ac17ce5374a) dm-12 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:12 sdo 8:224 active ready running
`- 1:0:0:12 sdao 66:128 active ready running
SY1_arch_disk_001 (350002ac17cca374a) dm-9 3PARdata,VV
size=150G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:9 sdl 8:176 active ready running
`- 1:0:0:9 sdal 66:80 active ready running
SY1_disknew_001 (350002ac4c566374a) dm-4 3PARdata,VV
size=60G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:4 sdg 8:96 active ready running
`- 1:0:0:4 sdag 66:0 active ready running
SCR_LOG2_disk_001 (350002acb5a39374a) dm-17 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:18 sdv 65:80 active ready running
`- 1:0:0:18 sdq 65:0 active ready running
MR2_arch_disk_001 (350002ac17cd7374a) dm-15 3PARdata,VV
size=20G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:14 sdr 65:16 active ready running
`- 1:0:0:14 sdaq 66:160 active ready running
SCR_LOG1_disk_001 (350002acb5a38374a) dm-19 3PARdata,VV
size=5.0G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:19 sdw 65:96 active ready running
`- 1:0:0:19 sdav 66:240 active ready running
MRC_data_disk_001 (350002ac15505374a) dm-2 3PARdata,VV
size=300G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:2 sde 8:64 active ready running
`- 1:0:0:2 sdae 65:224 active ready running
sap_disk_001 (350002ac1fc92374a) dm-13 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:16 sdt 65:48 active ready running
`- 1:0:0:16 sdas 66:192 active ready running
SY1_disk_001 (350002ac17cbd374a) dm-10 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:10 sdm 8:192 active ready running
`- 1:0:0:10 sdam 66:96 active ready running
SY1_interf_disk_001 (350002ac0ce50374a) dm-33 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:24 sdab 65:176 active ready running
`- 1:0:0:24 sdce 69:32 active ready running
MRC_log2_disk_001 (350002ac1551a374a) dm-71 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:0 sdc 8:32 active ready running
`- 1:0:0:0 sdac 65:192 active ready running
SY1_data_disk_002 (350002ac23826374a) dm-31 3PARdata,VV
size=2.6T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:22 sdz 65:144 active ready running
`- 1:0:0:22 sdcc 69:0 active ready running
MRC_log1_disk_001 (350002ac15519374a) dm-72 3PARdata,VV
size=10G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:1 sdd 8:48 active ready running
`- 1:0:0:1 sdad 65:208 active ready running
SCR_DATA_disk_001 (350002acb5a37374a) dm-23 3PARdata,VV
size=110G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:20 sdx 65:112 active ready running
`- 1:0:0:20 sdat 66:208 active ready running
SCR_disk_002 (350002ac19e73374a) dm-8 3PARdata,VV
size=50G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:8 sdk 8:160 active ready running
`- 1:0:0:8 sdak 66:64 active ready running
swap_disk_001 (350002ac155d0374a) dm-5 3PARdata,VV
size=224G features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 0:0:0:5 sdh 8:112 active ready running
`- 1:0:0:5 sdah 66:16 active ready running

Problem solved

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check mpmon
Fri Dec 15 11:39:25 2017 : INFO : UXMONmpmon is running now, pid=52191
mv: `/dev/null’ and `/dev/null’ are the same file
Fri Dec 15 11:39:26 2017 : INFO : UXMONmpmon end, pid=52191

Install HP Lights-Out Online Configuration Utility for Linux x86 for Suse Linux 8

HP Lights-Out Online Configuration Utility for Linux hponcfg-1.6.0-1.linux.rpm

HP Lights-Out Drivers and Agents for SUSE LINUX Enterprise Server 8 (x86) hprsm-7.5.0-169.sles8.i386.rpm

HP System Health Application and Insight Management Agents for SUSE LINUX Enterprise Server 8 (x86) hpasm-7.3.0-58.ul10.i386.rpm
Download and install the three packages shown above

root@linux:/tmp # rpm -ivh hponcfg-1.6.0-1.linux.rpm hprsm-7.5.0-169.sles8.i386.rpm hpasm-7.3.0-58.ul10.i386.rpm
hponcfg ##################################################
hpasm ##################################################
Please read the Licence Agreement for this software at

/opt/compaq/hpasm/hpasm.license

By not removing this package, you are accepting the terms
of the “License for HP Value Added Software”.
==============================================================================
NOTE: In order to activate the software contained in this package, you must
type ‘hpasm activate’ as ‘root’ user.
You may subsequently reset your agent configuration by typing
‘/etc/init.d/hpasm configure’ or ‘/etc/init.d/hpasm reconfigure’.
==============================================================================
The hpasm RPM has installed successfully.
hprsm ##################################################
cat: /opt/compaq/cpqci/driver/kernel.txt: No such file or directory
Looking for sources to build 2.4.21-278-smp
Rebuild successful.
..cpqci started.
Looking for sources to build 2.4.21-278-smp
Setting up hprsm sources for custom RPM
/opt/compaq/hprsm/etc/rebuild: line 35: /usr/src/packages/SPECS/hprsm.spec: No such file or directory
which: no rpmbuild in (/sbin:/bin:/usr/sbin:/usr/bin:/usr/X11R6/bin)
ls: /usr/src/packages/RPMS/i386/hprsm-*-custom.i386.rpm: No such file or directory
RPM build with /usr/src/packages/SPECS/hprsm.spec failed!
You must run ‘hpasm activate’ as ‘root’ user before
running the software agents contained in this package.
For proper configuration of the software agents in
this package, it is recommended that you run
‘/opt/compaq/hprsm/hprsm activate’ after you have
run ‘hpasm activate.’

The hprsm RPM has installed successfully.

===================================================================
NOTE: If you have not previously run the ‘hpasm activate’
command you must type ‘hpasm activate’ as ‘root’ user
to activate the agent software in this package.
===================================================================

Run hponcfg

root@linux:/tmp # hponcfg
Firmware Revision = 1.82 Device type = iLO Driver name = cpqci
HPONCFG RILOE II/iLO setup and configuration utility
Version 1.6 (c) Hewlett-Packard Company, 2005-2006

hponcfg -?
hponcfg -h
hponcfg -m minFw
hponcfg -r [-m minFw ]
hponcfg -w filename [-m minFw]
hponcfg -g [-m minFw]
hponcfg -f filename [-l filename] [-s namevaluepair] [-v] [-m minFw]

-h, –help Display this message
-? Display this message
-r, –reset Reset the RILOE II/iLO to factory default
-f, –file Get/Set RILOE II/iLO config from “filename”
-w, –writeconfig Write the RILOE II/iLO config to “filename”
-l, –log Log replies to “filename”
-v, –xmlverbose Display all the responses from RILOE II/iLO
-s, –substitute Substitute variables present in input config file
with values specified in “namevaluepairs”
-g, –get_hostinfo Get the Host information
-m, –minfwlevel Minimum firmware level

RHEL 5 server stopping boot process at message SELinux: Disabled at runtime. type=1404 audt(1358286571.639.2): selinux=0 auid=4294967295 ses=4294967295

Boot was stopping at the following message

SELinux: Disabled at runtime.
type=1404 audt(1358286571.639.2): selinux=0 auid=4294967295 ses=4294967295


To solve this problem, follow the document Why RHEL5 boot hangs after “SELinux: Disabled at runtime?

  • Disable console redirection.
  • Remove serial console entry from GRUB kernel line.

Rebooted the server
Edited grub line removing any mention of console, at the end of line inserted init=/bin/bash. Pressed then b to boot with these modified line
After getting prompt

# mount -o rw,remount /
# cp -p /etc/fstab /etc/fstab.20171211

Edited /etc/fstab and commented any filesystem that is not from the operating system

Rebooted the server

After the server is online, edited /boot/grub/menu.lst and removed all mention of console on kernel lines

root@linux:/boot/grub # vi menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/rootvg/rootlv
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
#splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu

serial –unit=0 –speed=115200
terminal –timeout=10 console serial

title Red Hat Enterprise Linux Server (2.6.18-408.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-408.el5 ro root=/dev/rootvg/rootlv rhgb noquiet crashkernel=256M@16M log_buf_len=3M elevator=noop nmi_watchdog=0
initrd /initrd-2.6.18-408.el5.img
title Red Hat Enterprise Linux Server (2.6.18-407.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-407.el5 ro root=/dev/rootvg/rootlv rhgb noquiet crashkernel=256M@16M log_buf_len=3M elevator=noop nmi_watchdog=0
initrd /initrd-2.6.18-407.el5.img
title Red Hat Enterprise Linux Server (2.6.18-398.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-398.el5 ro root=/dev/rootvg/rootlv rhgb noquiet crashkernel=256M@16M log_buf_len=3M elevator=noop nmi_watchdog=0
initrd /initrd-2.6.18-398.el5.img

Uncomment the filesystems you had just commented and then run fsck on them

root@linux:~ # vi /etc/fstab
/dev/rootvg/rootlv / ext4 defaults,nobarrier 1 1
/dev/rootvg/optlv /opt ext4 defaults,nobarrier 1 2
/dev/rootvg/tmplv /tmp ext4 defaults,nobarrier 1 2
/dev/rootvg/auditlv /audit ext4 defaults,nobarrier 1 2
/dev/rootvg/usrlv /usr ext4 defaults,nobarrier 1 2
/dev/rootvg/userlv /usr/users ext4 defaults,nobarrier 1 2
/dev/rootvg/varlv /var ext4 defaults,nobarrier 1 2
/dev/rootvg/crashlv /var/crash ext4 defaults,nobarrier 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/rootvg/swaplv swap swap defaults 0 0

## Files Systems Oracle ##
##/dev/mapper/softwarevg-softwarelv /usr/software ext4 defaults,nobarrier 1 2
##/dev/mapper/dcj200_vg-dcj200_data /usr/oradata/oradcj200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dcj200_vg-dcj200_arch /usr/oradata/archive/oradcj200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dit200_vg-dit200_data /usr/oradata/oradit200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dit200_vg-dit200_arch /usr/oradata/archive/oradit200 ext4 defaults,nobarrier 1 2
##/dev/mapper/drj200_vg-drj200_data /usr/oradata/oradrj200 ext4 defaults,nobarrier 1 2
##/dev/mapper/drj200_vg-drj200_arch /usr/oradata/archive/oradrj200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dsl200_vg-dsl200_data /usr/oradata/oradsl200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dsl200_vg-dsl200_arch /usr/oradata/archive/oradsl200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dvt200_vg-dvt200_data /usr/oradata/oradvt200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dvt200_vg-dvt200_arch /usr/oradata/archive/oradvt200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dbh200_vg-dbh200_data /usr/oradata/oradbh200 ext4 defaults,nobarrier 1 2
##/dev/mapper/dbh200_vg-dbh200_arch /usr/oradata/archive/oradbh200 ext4 defaults,nobarrier 1 2

Forcing to run fsck

fsck -t ext4 -y -f /dev/mapper/softwarevg-softwarelv
fsck -t ext4 -y -f /dev/mapper/dcj200_vg-dcj200_data
fsck -t ext4 -y -f /dev/mapper/dcj200_vg-dcj200_arch
fsck -t ext4 -y -f /dev/mapper/dit200_vg-dit200_data
fsck -t ext4 -y -f /dev/mapper/dit200_vg-dit200_arch
fsck -t ext4 -y -f /dev/mapper/drj200_vg-drj200_data
fsck -t ext4 -y -f /dev/mapper/drj200_vg-drj200_arch
fsck -t ext4 -y -f /dev/mapper/dsl200_vg-dsl200_data
fsck -t ext4 -y -f /dev/mapper/dsl200_vg-dsl200_arch
fsck -t ext4 -y -f /dev/mapper/dvt200_vg-dvt200_data
fsck -t ext4 -y -f /dev/mapper/dvt200_vg-dvt200_arch
fsck -t ext4 -y -f /dev/mapper/dbh200_vg-dbh200_data
fsck -t ext4 -y -f /dev/mapper/dbh200_vg-dbh200_arch

Mount filesystems

mount /usr/software
mount /usr/oradata/oradcj200
mount /usr/oradata/archive/oradcj200
mount /usr/oradata/oradit200
mount /usr/oradata/archive/oradit200
mount /usr/oradata/oradrj200
mount /usr/oradata/archive/oradrj200
mount /usr/oradata/oradsl200
mount /usr/oradata/archive/oradsl200
mount /usr/oradata/oradvt200
mount /usr/oradata/archive/oradvt200
mount /usr/oradata/oradbh200
mount /usr/oradata/archive/oradbh200

UXMON: Inode utilization of /var/opt exceeds 90 threshold

root@linux:/var/opt # df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vgroot-lv_var_opt
327680 308929 18751 95% /var/opt

I found a script at this page – How to Free Inode Usage?

Create a file and then run the script to find which directory are holding the most files

#!/bin/bash
# count_em – count files in all subdirectories under current directory.
echo ‘echo $(ls -a “$1” | wc -l) $1’ >/tmp/count_em_$$
chmod 700 /tmp/count_em_$$
find . -mount -type d -print0 | xargs -0 -n1 /tmp/count_em_$$ | sort -n
rm -f /tmp/count_em_$$

Most files are inside /var/opt/OV/log/OpC

root@linux:/var/opt/ # ./inodes.sh

457 ./osit/acf/log
599 ./newid
849 ./osit/linux/log/bdf.statistics
5066 ./erm/save
172726 ./OV/log/OpC

Deleted the files that I saw were unnecessary and it decreased inode consumption

root@linux:/var/opt # df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vgroot-lv_var_opt
327680 181506 146174 56% /var/opt

%d bloggers like this: