Category: Linux

Red Hat Enterprise Linux 5 with Cluster Suite Software is inquorate

One of the nodes of a Red Hat Enterprise Linux 5 with Cluster Suite software is inquorate

root@linux01:~ # clustat
Cluster Status for clinformatica @ Tue Jul 18 14:52:42 2017
Member Status: Quorate

Member Name ID Status
—— —- —- ——
linux01.heartbeat.local 1 Online, Local, rgmanager
linux02.heartbeat.local 2 Offline
/dev/mapper/qdisk0 0 Online, Quorum Disk

Service Name Owner (Last) State
——- —- —– —— —–
service:PCenterETL_41 linux01.heartbeat.local started
service:PCenterETL_42 (none) stopped

root@linux02:~ # clustat
Cluster Status for clinformatica @ Tue Jul 18 14:53:10 2017
Member Status: Inquorate

Member Name ID Status
—— —- —- ——
linux01.heartbeat.local 1 Offline
linux02.heartbeat.local 2 Online, Local
/dev/mapper/qdisk0 0 Offline

To troubleshoot the problem, install omping. It is part of EPEL

Download the package here: https://download.fedoraproject.org/pub/archive/epel/5/x86_64/omping-0.0.4-1.el5.x86_64.rpm
Install the package

root@linux01:/tmp # rpm -ivh omping-0.0.4-1.el5.x86_64.rpm
warning: omping-0.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 217521f6
Preparing… ########################################### [100%]
1:omping ########################################### [100%]

root@linux02:/tmp # rpm -ivh omping-0.0.4-1.el5.x86_64.rpm
warning: omping-0.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 217521f6
Preparing… ########################################### [100%]
1:omping ########################################### [100%]

Check which IP address the cluster uses

root@linux01:~ # grep clusternode /etc/cluster/cluster.conf | grep name

linux01-hb is 142.40.81.128
linux02-hb is 142.40.81.129

Run omping with the IP of the machine followed by the other node

root@linux01:~ # omping 142.40.81.128 142.40.81.129
142.40.81.129 : waiting for response msg
142.40.81.129 : waiting for response msg
142.40.81.129 : waiting for response msg
142.40.81.129 : waiting for response msg
142.40.81.129 : waiting for response msg
142.40.81.129 : waiting for response msg
142.40.81.129 : joined (S,G) = (*, 232.43.211.234), pinging
142.40.81.129 : unicast, seq=1, size=69 bytes, dist=0, time=0.246ms
142.40.81.129 : multicast, seq=1, size=69 bytes, dist=0, time=0.251ms

142.40.81.129 : multicast, seq=179, size=69 bytes, dist=0, time=0.269ms
142.40.81.129 : unicast, seq=180, size=69 bytes, dist=0, time=0.233ms
142.40.81.129 : multicast, seq=180, size=69 bytes, dist=0, time=0.239ms
142.40.81.129 : unicast, seq=181, size=69 bytes, dist=0, time=0.213ms
142.40.81.129 : multicast, seq=181, size=69 bytes, dist=0, time=0.219ms
142.40.81.129 : unicast, seq=182, size=69 bytes, dist=0, time=0.231ms
142.40.81.129 : multicast, seq=182, size=69 bytes, dist=0, time=0.236ms
142.40.81.129 : unicast, seq=183, size=69 bytes, dist=0, time=0.209ms
142.40.81.129 : multicast, seq=183, size=69 bytes, dist=0, time=0.286ms
142.40.81.129 : unicast, seq=184, size=69 bytes, dist=0, time=0.254ms
142.40.81.129 : unicast, seq=185, size=69 bytes, dist=0, time=0.176ms
142.40.81.129 : unicast, seq=186, size=69 bytes, dist=0, time=0.191ms
142.40.81.129 : unicast, seq=187, size=69 bytes, dist=0, time=0.291ms
142.40.81.129 : unicast, seq=188, size=69 bytes, dist=0, time=0.203ms
142.40.81.129 : unicast, seq=189, size=69 bytes, dist=0, time=0.199ms
142.40.81.129 : unicast, seq=190, size=69 bytes, dist=0, time=0.209ms
142.40.81.129 : unicast, seq=191, size=69 bytes, dist=0, time=0.145ms
142.40.81.129 : unicast, seq=192, size=69 bytes, dist=0, time=0.210ms
142.40.81.129 : unicast, seq=193, size=69 bytes, dist=0, time=0.281ms
142.40.81.129 : unicast, seq=194, size=69 bytes, dist=0, time=0.186ms
142.40.81.129 : unicast, seq=195, size=69 bytes, dist=0, time=0.195ms
142.40.81.129 : unicast, seq=196, size=69 bytes, dist=0, time=0.141ms
142.40.81.129 : unicast, seq=197, size=69 bytes, dist=0, time=0.205ms
142.40.81.129 : unicast, seq=198, size=69 bytes, dist=0, time=0.196ms
142.40.81.129 : unicast, seq=199, size=69 bytes, dist=0, time=0.179ms
142.40.81.129 : unicast, seq=200, size=69 bytes, dist=0, time=0.190ms

142.40.81.129 : unicast, xmt/rcv/%loss = 200/200/0%, min/avg/max/std-dev = 0.104/0.199/0.306/0.039
142.40.81.129 : multicast, xmt/rcv/%loss = 200/183/8%, min/avg/max/std-dev = 0.126/0.215/0.311/0.041

Running omping with the IP of the machine followed by the other node

root@linux02:~ # omping 142.40.81.129 142.40.81.128
142.40.81.128 : waiting for response msg
142.40.81.128 : joined (S,G) = (*, 232.43.211.234), pinging
142.40.81.128 : unicast, seq=1, size=69 bytes, dist=0, time=0.172ms
142.40.81.128 : multicast, seq=1, size=69 bytes, dist=0, time=0.287ms
142.40.81.128 : unicast, seq=2, size=69 bytes, dist=0, time=0.196ms
142.40.81.128 : multicast, seq=2, size=69 bytes, dist=0, time=0.252ms

142.40.81.128 : unicast, seq=184, size=69 bytes, dist=0, time=0.311ms
142.40.81.128 : multicast, seq=184, size=69 bytes, dist=0, time=0.367ms
142.40.81.128 : unicast, seq=185, size=69 bytes, dist=0, time=0.286ms
142.40.81.128 : multicast, seq=185, size=69 bytes, dist=0, time=0.338ms
142.40.81.128 : unicast, seq=186, size=69 bytes, dist=0, time=0.184ms
142.40.81.128 : unicast, seq=187, size=69 bytes, dist=0, time=0.193ms
142.40.81.128 : unicast, seq=188, size=69 bytes, dist=0, time=0.174ms
142.40.81.128 : unicast, seq=189, size=69 bytes, dist=0, time=0.192ms
142.40.81.128 : unicast, seq=190, size=69 bytes, dist=0, time=0.200ms
142.40.81.128 : unicast, seq=191, size=69 bytes, dist=0, time=0.241ms
142.40.81.128 : unicast, seq=192, size=69 bytes, dist=0, time=0.304ms
142.40.81.128 : unicast, seq=193, size=69 bytes, dist=0, time=0.259ms
142.40.81.128 : unicast, seq=194, size=69 bytes, dist=0, time=0.272ms
142.40.81.128 : unicast, seq=195, size=69 bytes, dist=0, time=0.246ms
142.40.81.128 : unicast, seq=196, size=69 bytes, dist=0, time=0.611ms
142.40.81.128 : unicast, seq=197, size=69 bytes, dist=0, time=0.208ms
142.40.81.128 : unicast, seq=198, size=69 bytes, dist=0, time=0.200ms
142.40.81.128 : unicast, seq=199, size=69 bytes, dist=0, time=0.194ms
142.40.81.128 : unicast, seq=200, size=69 bytes, dist=0, time=0.186ms
142.40.81.128 : unicast, seq=201, size=69 bytes, dist=0, time=0.190ms
142.40.81.128 : waiting for response msg
142.40.81.128 : server told us to stop

142.40.81.128 : unicast, xmt/rcv/%loss = 201/201/0%, min/avg/max/std-dev = 0.115/0.223/0.611/0.055
142.40.81.128 : multicast, xmt/rcv/%loss = 201/185/7%, min/avg/max/std-dev = 0.159/0.278/0.797/0.062

After a few minutes, the network switch blocked multicast communication.

Solved after reconfiguring network switch

How to Avoid a Split-Brain Scenario with Cisco Switches by Enabling Multicast Communication

-bash: /dev/null: Permission denied

After login I received a message showing permission problem on /dev/null

-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied
-bash: /dev/null: Permission denied

On a working server, it is a character device with rw permissions

root@workinglinux:~ # ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Jul 21 15:56 /dev/null

root@linux:~ # ls -l /dev/null
-rw-r–r– 1 root root 0 Jul 25 13:11 /dev/null

Remove the file and recreate it

root@linux:~ # rm -f /dev/null
root@linux:~ # mknod -m 666 /dev/null c 1 3

root@linux:~ # ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 Jul 25 13:13 /dev/null

rpm Fatal error, run database recovery

The rpm database got corrupted

root@linux:~ # rpm -qa
rpmdb: Thread/process 17308/47583269480000 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 – (-30974)
error: cannot open Packages database in /var/lib/rpm
rpmdb: Thread/process 17308/47583269480000 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm

Make a backup copy of /var/lib/rpm

root@linux:/var/lib # cp -pr rpm rpm.original

Listing the files inside /var/lib/rpm

root@linux:/var/lib/rpm # ls -al
total 127840
drwxr-xr-x. 2 root root 4096 Jul 15 21:08 .
drwxr-xr-x. 33 root root 4096 Sep 6 03:18 ..
-rw-r–r–. 1 root root 5619712 Jul 15 20:45 Basenames
-rw-r–r–. 1 root root 12288 Jul 15 20:45 Conflictname
-rw-r–r– 1 root root 73728 Sep 6 09:00 __db.001
-rw-r–r– 1 root root 237568 Sep 6 09:00 __db.002
-rw-r–r– 1 root root 1318912 Sep 6 09:00 __db.003
-rw-r–r– 1 root root 761856 Aug 31 14:30 __db.004
-rw-r–r–. 1 root root 3022848 Jul 15 20:45 Dirnames
-rw-r–r–. 1 root root 5558272 Jul 15 20:45 Filedigests
-rw-r–r–. 1 root root 24576 Jul 15 20:45 Group
-rw-r–r–. 1 root root 24576 Jul 15 20:45 Installtid
-rw-r–r–. 1 root root 40960 Jul 15 20:45 Name
-rw-r–r–. 1 root root 24576 Jul 15 20:45 Obsoletename
-rw-r–r–. 1 root root 112074752 Jul 15 20:45 Packages
-rw-r–r–. 1 root root 2465792 Jul 15 20:45 Providename
-rw-r–r–. 1 root root 1171456 Jul 15 20:45 Provideversion
-rw-r–r–. 1 root root 12288 Jul 10 2015 Pubkeys
-rw-r–r–. 1 root root 438272 Jul 15 20:45 Requirename
-rw-r–r–. 1 root root 270336 Jul 15 20:45 Requireversion
-rw-r–r–. 1 root root 0 Jul 10 2015 .rpm.lock
-rw-r–r–. 1 root root 167936 Jul 15 20:45 Sha1header
-rw-r–r–. 1 root root 81920 Jul 15 20:45 Sigmd5
-rw-r–r–. 1 root root 12288 Jul 15 20:45 Triggername

Removing all /var/lib/rpm/___db.XXX files

root@linux:/var/lib/rpm # rm -f /var/lib/rpm/__*

Listing the files again

root@linux:/var/lib/rpm # ls -la
total 125720
drwxr-xr-x. 2 root root 4096 Sep 6 09:07 .
drwxr-xr-x. 34 root root 4096 Sep 6 09:07 ..
-rw-r–r–. 1 root root 5619712 Jul 15 20:45 Basenames
-rw-r–r–. 1 root root 12288 Jul 15 20:45 Conflictname
-rw-r–r–. 1 root root 3022848 Jul 15 20:45 Dirnames
-rw-r–r–. 1 root root 5558272 Jul 15 20:45 Filedigests
-rw-r–r–. 1 root root 24576 Jul 15 20:45 Group
-rw-r–r–. 1 root root 24576 Jul 15 20:45 Installtid
-rw-r–r–. 1 root root 40960 Jul 15 20:45 Name
-rw-r–r–. 1 root root 24576 Jul 15 20:45 Obsoletename
-rw-r–r–. 1 root root 112074752 Jul 15 20:45 Packages
-rw-r–r–. 1 root root 2465792 Jul 15 20:45 Providename
-rw-r–r–. 1 root root 1171456 Jul 15 20:45 Provideversion
-rw-r–r–. 1 root root 12288 Jul 10 2015 Pubkeys
-rw-r–r–. 1 root root 438272 Jul 15 20:45 Requirename
-rw-r–r–. 1 root root 270336 Jul 15 20:45 Requireversion
-rw-r–r–. 1 root root 0 Jul 10 2015 .rpm.lock
-rw-r–r–. 1 root root 167936 Jul 15 20:45 Sha1header
-rw-r–r–. 1 root root 81920 Jul 15 20:45 Sigmd5
-rw-r–r–. 1 root root 12288 Jul 15 20:45 Triggername

Rebuilding database indices from installed package headers

rpm {–initdb|–rebuilddb} [-v] [–dbpath DIRECTORY] [–root DIRECTORY]

Use –initdb to create a new database if one doesn’t already exist (existing database is not overwritten), use –rebuilddb to rebuild the
database indices from the installed package headers.

root@linux:/var/lib/rpm # rpm –rebuilddb

rpm doesn’t return any errors.

root@linux:/var/lib/rpm # rpm -qa | wc -l
927

Shrinking ext4 filesystem

I’m creating a new volume group with 3 10GB disks

root@linux:~ # pvcreate /dev/sdc
Physical volume “/dev/sdc” successfully created
root@linux:~ # pvcreate /dev/sdd
Physical volume “/dev/sdd” successfully created
root@linux:~ # pvcreate /dev/sde
Physical volume “/dev/sde” successfully created

root@linux:~ # vgcreate appvg /dev/sdc /dev/sdd /dev/sde
Volume group “appvg” successfully created

root@linux:~ # vgs appvg
VG #PV #LV #SN Attr VSize VFree
appvg 3 0 0 wz–n- 29.99g 29.99g

Creating a new 25GB logical volume

root@linux:~ # lvcreate -L 25g -n applv appvg
Logical volume “applv” created.

Creating a filesystem

root@linux:~ # mkfs -t ext4 /dev/appvg/applv
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1638400 inodes, 6553600 blocks
327680 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
200 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Mounting the filesystem

root@linux:~ # mkdir /app

root@linux:~ # mount /dev/appvg/applv /app

root@linux:~ # df -hP /app
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/appvg-applv 25G 44M 24G 1% /app

I will remove the disk /dev/sde from the volume group. 25GB logical volume will shrink to 20GB.

Checking which disks the logical volume is using

root@linux:~ # lvdisplay -m /dev/mapper/appvg-applv
— Logical volume —
LV Path /dev/appvg/applv
LV Name applv
VG Name appvg
LV UUID 9tofzm-1zBU-DfbM-nmws-lqj7-QULb-U4Reqh
LV Write Access read/write
LV Creation host, time centos6.setaoffice.com, 2017-06-29 16:53:18 -0300
LV Status available
# open 1
LV Size 25.00 GiB
Current LE 6400
Segments 3
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:6

— Segments —
Logical extents 0 to 2558:
Type linear
Physical volume /dev/sdc
Physical extents 0 to 2558

Logical extents 2559 to 5117:
Type linear
Physical volume /dev/sdd
Physical extents 0 to 2558

Logical extents 5118 to 6399:
Type linear
Physical volume /dev/sde
Physical extents 0 to 1281

Checking Total PE to calculate logical volume

root@linux:~ # pvdisplay /dev/sdc
— Physical volume —
PV Name /dev/sdc
VG Name appvg
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2559
Free PE 0
Allocated PE 2559
PV UUID ZnR27c-LPaE-0ZQa-t0Hc-yWLf-v06d-et7fb5

root@linux:~ # pvdisplay /dev/sdd
— Physical volume —
PV Name /dev/sdd
VG Name appvg
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2559
Free PE 0
Allocated PE 2559
PV UUID OZY6w8-fUq6-wbPE-ILcr-LwOP-AeaS-j2qd9E

Resizing logical volume

root@linux:~ # lvresize -l 5118 -r /dev/appvg/applv
fsck from util-linux-ng 2.17.2
/dev/mapper/appvg-applv: 11/1638400 files (0.0% non-contiguous), 146849/6553600 blocks
resize2fs 1.41.12 (17-May-2010)
Resizing the filesystem on /dev/mapper/appvg-applv to 5240832 (4k) blocks.
The filesystem on /dev/mapper/appvg-applv is now 5240832 blocks long.

Size of logical volume appvg/applv changed from 25.00 GiB (6400 extents) to 19.99 GiB (5118 extents).
Logical volume applv successfully resized.

Checking which disks the logical volume is using

root@linux:~ # lvdisplay -m /dev/mapper/appvg-applv
— Logical volume —
LV Path /dev/appvg/applv
LV Name applv
VG Name appvg
LV UUID 9tofzm-1zBU-DfbM-nmws-lqj7-QULb-U4Reqh
LV Write Access read/write
LV Creation host, time centos6.setaoffice.com, 2017-06-29 16:53:18 -0300
LV Status available
# open 0
LV Size 19.99 GiB
Current LE 5118
Segments 2
Allocation inherit
Read ahead sectors auto
– currently set to 256
Block device 253:6

— Segments —
Logical extents 0 to 2558:
Type linear
Physical volume /dev/sdc
Physical extents 0 to 2558

Logical extents 2559 to 5117:
Type linear
Physical volume /dev/sdd
Physical extents 0 to 2558

Removing disk from volume group and label from physical volume

root@linux:~ # vgreduce appvg /dev/sde
Removed “/dev/sde” from volume group “appvg”

root@linux:~ # pvremove /dev/sde
Labels on physical volume “/dev/sde” successfully wiped

Red Hat Linux guest on VMware: Bringing up interface eth0: vmxnet3 device eth0 does not seem to be present, delaying initialization

I had a problem where I after a Red Hat Linux guest running on VMware was showing the following message on start up

Bringing up interface eth0: vmxnet3 device eth0 does not seem to be present, delaying initialization

To solve this problem, remove VMware Tools and reinstall it

UXMON: slave eno49 of bonding device bond0 in Red Hat Enterprise Linux 7

ATTENTION, RMC LEVEL 1 AGENT: This ticket will be automatically worked by the Automation Bus. Pls. ensure your Ticket List/View includes the “Assignee” column, monitor this ticket until the user “ABOPERATOR” is no longer assigned, BEFORE you start work on this ticket.
Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : major
OM Server Time: 2017-03-07 09:50:03
Message : UXMON: slave eno49 of bonding device bond0
Msg Group : OS
Application : bondmon
Object : bond
Event Type :
not_found

Instance Name :
not_found

Instruction : The ‘cat /sys/class/net/bondX/slave_ethY/operstate’ command shows detail

Please check /var/opt/OV/log/OpC/bond_mon.log for more details

Running UXMONbroker we see that there are some files that don’t exist

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check bondmon
Mon Mar 13 15:47:35 2017 : INFO : UXMONbondmon is running now, pid=17458
cat: /sys/class/net/bond0/slave_eno49/operstate: No such file or directory
Mon Mar 13 15:47:35 2017 : Major: slave eno49 of bonding device bond0
cat: /sys/class/net/bond0/slave_eno50/operstate: No such file or directory
Mon Mar 13 15:47:35 2017 : Major: slave eno50 of bonding device bond0
mv: ‘/dev/null’ and ‘/dev/null’ are the same file
Mon Mar 13 15:47:35 2017 : INFO : UXMONbondmon end, pid=17458

root@linux:~ # ls -l /sys/class/net/bond0/
total 0
-r–r–r– 1 root root 4096 Mar 2 13:35 addr_assign_type
-r–r–r– 1 root root 4096 Mar 7 17:38 address
-r–r–r– 1 root root 4096 Mar 7 17:38 addr_len
drwxr-xr-x 2 root root 0 Mar 2 13:35 bonding
-r–r–r– 1 root root 4096 Mar 7 17:38 broadcast
-rw-r–r– 1 root root 4096 Mar 7 17:38 carrier
-r–r–r– 1 root root 4096 Mar 7 17:38 carrier_changes
-r–r–r– 1 root root 4096 Mar 2 13:35 dev_id
-r–r–r– 1 root root 4096 Mar 7 17:38 dev_port
-r–r–r– 1 root root 4096 Mar 7 17:38 dormant
-r–r–r– 1 root root 4096 Mar 7 17:38 duplex
-rw-r–r– 1 root root 4096 Mar 7 17:38 flags
-rw-r–r– 1 root root 4096 Mar 7 17:38 gro_flush_timeout
-rw-r–r– 1 root root 4096 Mar 7 17:38 ifalias
-r–r–r– 1 root root 4096 Mar 2 13:35 ifindex
-r–r–r– 1 root root 4096 Mar 2 13:35 iflink
-r–r–r– 1 root root 4096 Mar 7 17:38 link_mode
lrwxrwxrwx 1 root root 0 Mar 7 17:38 lower_eno49 -> ../../../pci0000:00/0000:00:02.0/0000:06:00.0/net/eno49
lrwxrwxrwx 1 root root 0 Mar 7 17:38 lower_eno50 -> ../../../pci0000:00/0000:00:02.0/0000:06:00.1/net/eno50
-rw-r–r– 1 root root 4096 Mar 7 17:38 mtu
-rw-r–r– 1 root root 4096 Mar 7 17:38 netdev_group
-r–r–r– 1 root root 4096 Mar 7 17:38 operstate
-r–r–r– 1 root root 4096 Mar 2 13:35 phys_port_id
drwxr-xr-x 2 root root 0 Mar 2 13:41 power
drwxr-xr-x 34 root root 0 Mar 2 13:35 queues
-r–r–r– 1 root root 4096 Mar 7 17:38 speed
drwxr-xr-x 2 root root 0 Mar 2 13:41 statistics
lrwxrwxrwx 1 root root 0 Mar 2 13:35 subsystem -> ../../../../class/net
-rw-r–r– 1 root root 4096 Mar 7 17:38 tx_queue_len
-r–r–r– 1 root root 4096 Mar 2 13:35 type
-rw-r–r– 1 root root 4096 Mar 2 13:35 uevent

This is a Red Hat Enterprise Linux 7. Current HPOM have a bug that is querying the wrong file

root@linux:~ # cat /etc/*release
NAME=”Red Hat Enterprise Linux Server”
VERSION=”7.3 (Maipo)”
ID=”rhel”
ID_LIKE=”fedora”
VERSION_ID=”7.3″
PRETTY_NAME=”Red Hat Enterprise Linux”
ANSI_COLOR=”0;31″
CPE_NAME=”cpe:/o:redhat:enterprise_linux:7.3:GA:server”
HOME_URL=”https://www.redhat.com/”
BUG_REPORT_URL=”https://bugzilla.redhat.com/”

REDHAT_BUGZILLA_PRODUCT=”Red Hat Enterprise Linux 7″
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT=”Red Hat Enterprise Linux”
REDHAT_SUPPORT_PRODUCT_VERSION=”7.3″
Red Hat Enterprise Linux Server release 7.3 (Maipo)
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Copy the bond_mon.cfg configuration file and disable this module

cp -p /var/opt/OV/bin/instrumentation/bond_mon.cfg /var/opt/OV/conf/OpC
vi /var/opt/OV/conf/OpC/bond_mon.cfg
disable = yes

root@linux:~ # /var/opt/OV/bin/instrumentation/UXMONbroker -check bondmon
root@linux:~ #

One path missing in disk map on multipath device

Showing a particular case:

The disk mpath5 was only showing one path

root@linux:~ # multipath -ll mpath5
mpath5 (350002ac19430374a) dm-17 3PARdata,VV
[size=47G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 2:0:0:1 sdh 8:112 [active][ready]

The disk used by operating system is cciss/c0d0

root@linux:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/cciss/c0d0p3 vg00 lvm2 a– 269.47G 203.28G
/dev/mpath/350002ac19429374a vgapp lvm2 a– 100.00G 0
/dev/mpath/350002ac1942c374a vgapp lvm2 a– 20.00G 0
/dev/mpath/350002ac1942e374a vgapp lvm2 a– 75.00G 0
/dev/mpath/350002ac1942f374a vgapp lvm2 a– 158.00G 0
/dev/mpath/350002ac19430374a vgapp lvm2 a– 47.00G 996.00M
/dev/mpath/350002ac22869374a vgapp lvm2 a– 100.00G 0
/dev/mpath/350002ac2286a374a vgapp lvm2 a– 40.00G 0

Listing the SCSI devices. sda through sdn are used

root@linux:~ # lsscsi
[1:0:0:1] disk 3PARdata VV 3213 /dev/sda
[1:0:0:2] disk 3PARdata VV 3213 /dev/sdb
[1:0:0:3] disk 3PARdata VV 3213 /dev/sdc
[1:0:0:4] disk 3PARdata VV 3213 /dev/sdd
[1:0:0:5] disk 3PARdata VV 3213 /dev/sde
[1:0:0:6] disk 3PARdata VV 3213 /dev/sdf
[1:0:0:7] disk 3PARdata VV 3213 /dev/sdg
[1:0:0:254] enclosu 3PARdata SES 3213 –
[2:0:0:1] disk 3PARdata VV 3213 /dev/sdh
[2:0:0:2] disk 3PARdata VV 3213 /dev/sdi
[2:0:0:3] disk 3PARdata VV 3213 /dev/sdj
[2:0:0:4] disk 3PARdata VV 3213 /dev/sdk
[2:0:0:5] disk 3PARdata VV 3213 /dev/sdl
[2:0:0:6] disk 3PARdata VV 3213 /dev/sdm
[2:0:0:7] disk 3PARdata VV 3213 /dev/sdn
[2:0:0:254] enclosu 3PARdata SES 3213 –

Checking /etc/multipath.conf. sda was being blacklisted. Commented the line

root@linux:~ # grep -v ^# /etc/multipath.conf

blacklist {
devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”
devnode “^hd[a-z][[0-9]*]”
devnode “^hd[a-z]”
#devnode “^sda$”
}

defaults {
user_friendly_names yes
}

Running multipath -v3

root@linux:~ # multipath -v3

Checking disk mpath5

root@linux:~ # multipath -ll mpath5
mpath5 (350002ac19430374a) dm-17 3PARdata,VV
[size=47G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
\_ 1:0:0:1 sda 8:0 [active][ready]
\_ 2:0:0:1 sdh 8:112 [active][ready]

Remove ^M (Control-M) characters from a file in Unix

When checking my /etc/multipath.conf file, there was a lot of hidden characters ^M at the end of lines

root@linux:~ # cat -vet /etc/multipath.conf

To remove them, use tr and redirect the output to a new file.

root@linux:~ # tr -d ‘\015’ < /tmp/multipath.conf

Check the file and if everything is okay, replace the original file

root@linux:~ # tr -d ‘\015’ < /tmp/multipath.conf > /tmp/multipath.conf.new
root@linux:~ # cat -vet /tmp/multipath2.conf.new
root@linux:~ # mv /tmp/multipath2.conf.new /tmp/multipath.conf

LVM: Please specify number of stripes (-i) and stripesize (-I)

I tried to resize a logical volume but showed a message to specify stripes and stripesize

root@linux:~ # lvresize -L +200g -r /dev/vgER0data/lv_sapdata1
Please specify number of stripes (-i) and stripesize (-I)
Run `lvresize –help’ for more information.

Showing stripes and stripesize

root@linux:~ # lvdisplay /dev/vgER0data/lv_sapdata1 -m
— Logical volume —
LV Path /dev/vgER0data/lv_sapdata1
LV Name lv_sapdata1
VG Name vgER0data
LV UUID h2WKAp-h16f-O3Bo-QpEz-fTiW-s3Hr-K7W9sb
LV Write Access read/write
LV Creation host, time cavadb77, 2015-09-24 11:31:33 -0300
LV Status available
# open 1
LV Size 2.99 TiB
Current LE 783355
Segments 2
Allocation inherit
Read ahead sectors auto
– currently set to 32768
Block device 253:423

— Segments —
Logical extents 0 to 655359:
Type striped
Stripes 2
Stripe size 4.00 MiB
Stripe 0:
Physical volume /dev/mapper/ER0_data_disk_001p1
Physical extents 0 to 327679
Stripe 1:
Physical volume /dev/mapper/ER0_data_disk_002p1
Physical extents 0 to 327679

Logical extents 655360 to 783354:
Type linear
Physical volume /dev/mapper/ER0_data_disk_003
Physical extents 0 to 127994

To solve the problem in this case it was necessary to add a new LUN and resize the logical volume. The segment where it is linear, was kept this way. There is no way to convert

root@linux:~ # lvresize -L +100g -r -i2 -I4M /dev/mapper/vgER0data-lv_sapdata1
Size of logical volume vgER0data/lv_sapdata1 changed from 2.99 TiB (783355 extents) to 3.09 TiB (808955 extents).
Logical volume lv_sapdata1 successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vgER0data-lv_sapdata1 is mounted on /oracle/ER0/sapdata1; on-line resizing required
old desc_blocks = 192, new_desc_blocks = 198
Performing an on-line resize of /dev/mapper/vgER0data-lv_sapdata1 to 828369920 (4k) blocks.
The filesystem on /dev/mapper/vgER0data-lv_sapdata1 is now 828369920 blocks long.

 

VMware guest server taking a long time to finish boot stuck at Bringing up interface

Rebooted a server and saw some messages about VM communication interface showing errors

Starting VMware Tools services in the virtual machine
Switching to guest configuration: OK
Paravirtual SCSI module: OK
Guest memory manager: OK
VM communication interface: FAILED
VM communication interface socket family: FAILED
File system sync driver: OK
Guest operating system daemon: OK
FAILED

Long time at the message Bringing up interface eth0
vmware_bringing
Updated VMware Tools VMware Tools does not start after updating the kernel

Had to changed the driver for network interface for the server to finish the boot sequence