Using smsconnectsc to connect to another System Controller

Use smsconnectsc to connect to the other System Controller when it is inacessible by its IP address

sms-svc@sc:~ $ smsconnectsc
Attempting to determine remote SC power state. This may take some time.
Attempting to get TTY connection state. This may take some time.
TTY connection is OFF.
About to connect to other SC. Do you want to continue (yes/no)? yes
Attempting to set TTY connection state. This may take some time.
connected

HPUX: vmunix: Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the target path (class = tgtpath, instance = 27)

Looking at the logfile /var/adm/syslog/syslog.log I saw the messages

Mar 26 17:06:17 hpux vmunix: Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the target path (class = tgtpath, instance = 37) The lun path is (class = lunpath, instance 33).Run ‘scsimgr replace_wwid’ command to validate the change
Mar 26 17:06:47 hpux vmunix: Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the target path (class = tgtpath, instance = 27) The lun path is (class = lunpath, instance 25).Run ‘scsimgr replace_wwid’ command to validate the change

Running the command to validate the change

root@hpux:~ # scsimgr replace_wwid -C lunpath -I 27
scsimgr:WARNING: Performing replace_wwid on the resource may have some impact on system operation.
Do you really want to replace? (y/[n])? y
Binding of LUN path 0/2/0/0/0/0.0x5d8d385c1a8e4000.0x0 with new LUN validated successfully

root@hpux:~ # scsimgr -f replace_wwid -C lunpath -I 37
Binding of LUN path 0/4/0/0/0/1.0x5d8d385c1a8e4010.0x0 with new LUN validated successfully

HP-UX: UXMON:Critical multipath error detected. Please see /var/opt/OV/log/OpC/scsi_mon.log for details.

I received this problem where the disk was showing some multipath erros on an HP-UX server

UXMON:Critical multipath error detected. Please see /var/opt/OV/log/OpC/scsi_mon.log for details.

Node : hpux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : critical
OM Server Time: 2016-05-24 11:35:09
Message : UXMON:Critical multipath error detected. Please see /var/opt/OV/log/OpC/scsi_mon.log for details.
Msg Group : OS
Application : scsimon
Object : No
Event Type :
Instance Name :
Instruction : No

root@hpux:~ # cat /var/opt/OV/log/OpC/scsi_mon.log
Tue May 24 12:35:07 2016 : Critical /dev/rdisk/disk670 has failed lunpaths! Please check with scsimgr -p lun_map -D /dev/rdisk/disk670

root@hpux:~ # scsimgr -p lun_map -D /dev/rdisk/disk670
lunpath:647:38/0/0/2/0/0/0.0x50060e800574f200.0x405c000000000000:fibre_channel:FAILED:FAILED
lunpath:632:38/0/0/2/0/0/1.0x50060e800574f210.0x405c000000000000:fibre_channel:FAILED:FAILED
lunpath:617:36/0/0/2/0/0/0.0x50060e800574f200.0x405c000000000000:fibre_channel:FAILED:FAILED
lunpath:602:36/0/0/2/0/0/1.0x50060e800574f210.0x405c000000000000:fibre_channel:FAILED:FAILED
lunpath:33:36/0/0/2/0/0/0.0x50060e800574f200.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:35:36/0/0/2/0/0/1.0x50060e800574f210.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:37:38/0/0/2/0/0/0.0x50060e800574f200.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:39:38/0/0/2/0/0/1.0x50060e800574f210.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE

Removing invalid paths

root@hpux:~ # rmsf -H 38/0/0/2/0/0/0.0x50060e800574f200.0x405c000000000000
root@hpux:~ # rmsf -H 38/0/0/2/0/0/1.0x50060e800574f210.0x405c000000000000
root@hpux:~ # rmsf -H 36/0/0/2/0/0/0.0x50060e800574f200.0x405c000000000000
root@hpux:~ # rmsf -H 36/0/0/2/0/0/1.0x50060e800574f210.0x405c000000000000

Checking LUN paths

root@hpux:~ # scsimgr -p lun_map -D /dev/rdisk/disk670
lunpath:33:36/0/0/2/0/0/0.0x50060e800574f200.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:35:36/0/0/2/0/0/1.0x50060e800574f210.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:37:38/0/0/2/0/0/0.0x50060e800574f200.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:39:38/0/0/2/0/0/1.0x50060e800574f210.0x40a5000000000000:fibre_channel:ACTIVE:ACTIVE

Checking another disk

root@hpux:~ # scsimgr -p lun_map -D /dev/rdisk/disk31
lunpath:61:0/2/1/0.0x21230002ac001673.0x4001000000000000:fibre_channel: FAILED:AUTH_FAILED
lunpath:62:0/2/1/0.0x21230002ac001673.0x4002000000000000:fibre_channel:ACTIVE:ACTIVE
lunpath:71:0/5/1/0.0x20240002ac001673.0x4001000000000000:fibre_channel: FAILED:AUTH_FAILED
lunpath:72:0/5/1/0.0x20240002ac001673.0x4002000000000000:fibre_channel:ACTIVE:ACTIVE

root@hpux:~ # ioscan -m lun /dev/rdisk/disk31
Class I Lun H/W Path Driver S/W State H/W Type Health Description
=======================================================================
disk 31 64000/0xfa00/0x8c esdisk CLAIMED DEVICE limited 3PARdataVV
0/2/1/0.0x21230002ac001673.0x4001000000000000
0/2/1/0.0x21230002ac001673.0x4002000000000000
0/5/1/0.0x20240002ac001673.0x4001000000000000
0/5/1/0.0x20240002ac001673.0x4002000000000000
/dev/disk/disk31 /dev/rdisk/disk31

2 Failed LUN paths

root@hpux:~ # scsimgr get_info -D /dev/rdisk/disk31|more

STATUS INFORMATION FOR LUN : /dev/rdisk/disk31

Generic Status Information

SCSI services internal state = ONLINE
Device type = Direct_Access
EVPD page 0x83 description code = 1
EVPD page 0x83 description association = 0
EVPD page 0x83 description type = 3
World Wide Identifier (WWID) = 0x50002ac0031a1673
Serial number = ” 1405747″
Vendor id = “3PARdata”
Product id = “VV ”
Product revision = “3131”
Other properties = “”
SPC protocol revision = 6
Open count (includes chr/blk/pass-thru/class) = 1
Raw open count (includes class/pass-thru) = 0
Pass-thru opens = 0
LUN path count = 4
Active LUN paths = 2
Standby LUN paths = 0
Failed LUN paths = 2
Maximum I/O size allowed = 2097152
Preferred I/O size = 2097152
Outstanding I/Os = 0
I/O load balance policy = round_robin
Path fail threshold time period = 0
Transient time period = 120
Tracing buffer size = 1024
LUN Path used when policy is path_lockdown = NA
LUN access type = NA
Asymmetric logical unit access supported = No
Asymmetric states supported = NA
Preferred paths reported by device = No
Preferred LUN paths = 0

Driver esdisk Status Information :

Capacity in number of blocks = 213909504
Block size in bytes = 512
Number of active IOs = 0
Special properties =
Maximum number of IO retries = 45
IO transfer timeout in secs = 30
FORMAT command timeout in secs = 86400
START UNIT command timeout in secs = 60
Timeout in secs before starting failing IO = 120
IO infinite retries = false

Validating disk paths for disk31

root@hpux:~ # scsimgr -f replace_wwid -D /dev/rdisk/disk31
scsimgr: Successfully validated binding of LUN paths with new LUN.

The invalid paths were removed

root@hpux:~ # ioscan -m lun /dev/rdisk/disk31
Class I Lun H/W Path Driver S/W State H/W Type Health Description
======================================================================
disk 31 64000/0xfa00/0x8c esdisk CLAIMED DEVICE online 3PARdataVV
0/2/1/0.0x21230002ac001673.0x4002000000000000
0/5/1/0.0x20240002ac001673.0x4002000000000000
/dev/disk/disk31 /dev/rdisk/disk31

HP-UX: LVM – Failure possibly caused by PVG-Strict or Distributed allocation policies

Tried to create a logical volume and HP-uX gave an error message

root@hpux:~ # lvcreate -s g -D y -r N -L 400000 -n lvsapdata8 /dev/vgLP0data
Warning: The “-r” option has been ignored as it is not supported
for volume group version 2.0 or higher
Logical volume “/dev/vgLP0data/lvsapdata8” has been successfully created with
character device “/dev/vgLP0data/rlvsapdata8”.
lvcreate: Not enough free physical extents available.
Logical volume “/dev/vgLP0data/lvsapdata8” could not be extended.
Failure possibly caused by PVG-Strict or Distributed allocation policies.

The problem was the combination of strict allocation policy and distributed allocation. Since I had 7 disks in the volume group and a new disk was added, this logical volume creation was not meeting the options set

-s strict Set the strict allocation policy. Mirror copies
of a logical extent can be allocated to share or
not share the same physical volume or physical
volume group. strict can have one of the
following values:

y Set a strict allocation policy. Mirrors of
a logical extent cannot share the same
physical volume. This is the default.

g Set a PVG-strict allocation policy.
Mirrors of a logical extent cannot share
the same physical volume group. A PVG-
strict allocation policy cannot be set on a
logical volume in a volume group that does
not have a physical volume group defined.

n Do not set a strict or PVG-strict
allocation policy. Mirrors of a logical
extent can share the same physical volume.

-D distributed Set the distributed allocation policy.
distributed can have one of the following
values:

y Turn on distributed allocation.

n Turn off distributed allocation. This is
the default.

When the distributed allocation policy is turned
on, only one free extent is allocated from the
first available physical volume. The next free
extent is allocated from the next available
physical volume. Allocation of free extents
proceeds in round-robin order on the list of
available physical volumes.

When the distributed allocation policy is turned
off, all available free extents are allocated
from each available physical volume before
proceeding to the next available physical
volume. This is the default.

The distributed allocation policy REQUIRES the
PVG-strict allocation policy (-s g) to ensure
that mirrors of distributed extents do not
overlap (for maximum availability).

lvcreate(1M) will obtain the list of available
physical volumes from /etc/lvmpvg. See
vgextend(1M) for more information on physical
volume groups and /etc/lvmpvg.

When a logical volume with distributed extents
is mirrored, the resulting layout is commonly
referred to as EXTENT-BASED MIRRORED STRIPES.

Note that EXTENT-BASED MIRRORED STRIPES can be
created without the distributed allocation
policy by adding one extent at a time to the
desired physical volumes through lvextend(1M).

The distributed allocation policy is
incompatible with the striped scheduling policy
(-i stripes) and the contiguous allocation
policy (-C y).

The lvchange(1M) command can be used to assign
the distributed allocation policy to an existing
logical volume.

See lvdisplay(1M) for display values.

See EXAMPLES.

HP-UX: Disk added to Volume Group but showing different Total PE and Free PE than expected

After I added a disk to LVM

root@hpux:~ # pvcreate /dev/rdisk/disk670
Physical volume “/dev/rdisk/disk670” has been successfully created.

Disk added but it shows wrong total and free PE

root@hpux:~ # vgextend -g pvgLP0data vgLP0data /dev/disk/disk670
Volume group “vgLP0data” has been successfully extended.
Volume Group configuration for /dev/vgLP0data has been saved in /etc/lvmconf/vgLP0data.conf
vlunx045:/root# vgdisplay -v vgLP0data
— Volume groups —
VG Name /dev/vgLP0data
VG Write Access read/write
VG Status available, exclusive
Max LV 2047
Cur LV 8
Open LV 8
Cur Snapshot LV 0
Max PV 2048
Cur PV 8
Act PV 8
Max PE per PV 447994
VGDA 16
PE Size (Mbytes) 32
Unshare unit size (Kbytes) 1024
Total PE 447994
Alloc PE 447986
Current pre-allocated PE 0
Free PE 8
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 2.2
VG Max Size 14335819m
VG Max Extents 447994
Cur Snapshot Capacity 0p
Max Snapshot Capacity 14335819m

— Logical volumes —
LV Name /dev/vgLP0data/lvsapdata1
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata2
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata3
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata4
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata5
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata6
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata7
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata8
LV Status available/syncd
LV Size (Mbytes) 0
Current LE 0
Allocated PE 0
Used PV 0

— Physical volumes —
PV Name /dev/disk/disk949
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk950
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk951
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk952
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk953
PV Status available
Total PE 63999
Free PE 2
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk954
PV Status available
Total PE 63999
Free PE 2
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk955
PV Status available
Total PE 63999
Free PE 3
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk670
PV Status available
Total PE 1
Free PE 1
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

— Physical volume groups —
PVG Name pvgLP0data
PV Name /dev/disk/disk949
PV Name /dev/disk/disk950
PV Name /dev/disk/disk951
PV Name /dev/disk/disk952
PV Name /dev/disk/disk953
PV Name /dev/disk/disk954
PV Name /dev/disk/disk955
PV Name /dev/disk/disk670

We need to change VG Max Size. But first, let’s remove the disk from the volume group

root@hpux:~ # vgreduce vgLP0data /dev/disk/disk670
Physical volume “/dev/disk/disk670” has been successfully deleted from
physical volume group “pvgLP0data”.
Volume group “vgLP0data” has been successfully reduced.
Volume Group configuration for /dev/vgLP0data has been saved in /etc/lvmconf/vgLP0data.conf

Then run vgmodify to change the VG Max Size for the volume group. This volume group version is 2.2 so we use the following options:

root@hpux:~ # vgmodify -r -a -S 20t vgLP0data
Reconfiguration of physical volume “/dev/disk/disk949” for the
requested maximum volume group size 20971520 MB succeeded.
Previous number of extents: 63999
Number of extents after reconfiguration: 63999
Physical volume “/dev/disk/disk949” was changed.

Volume Group configuration for /dev/vgLP0data has been saved.

Reconfiguration of physical volume “/dev/disk/disk950” for the
requested maximum volume group size 20971520 MB succeeded.
Previous number of extents: 63999
Number of extents after reconfiguration: 63999
Physical volume “/dev/disk/disk950” was changed.

Volume Group configuration for /dev/vgLP0data has been saved.

Reconfiguration of physical volume “/dev/disk/disk951” for the
requested maximum volume group size 20971520 MB succeeded.
Previous number of extents: 63999
Number of extents after reconfiguration: 63999
Physical volume “/dev/disk/disk951” was changed.

Volume Group configuration for /dev/vgLP0data has been saved.

Reconfiguration of physical volume “/dev/disk/disk952” for the
requested maximum volume group size 20971520 MB succeeded.
Previous number of extents: 63999
Number of extents after reconfiguration: 63999
Physical volume “/dev/disk/disk952” was changed.

Volume Group configuration for /dev/vgLP0data has been saved.

Reconfiguration of physical volume “/dev/disk/disk953” for the
requested maximum volume group size 20971520 MB succeeded.
Previous number of extents: 63999
Number of extents after reconfiguration: 63999
Physical volume “/dev/disk/disk953” was changed.

Volume Group configuration for /dev/vgLP0data has been saved.

Reconfiguration of physical volume “/dev/disk/disk954” for the
requested maximum volume group size 20971520 MB succeeded.
Previous number of extents: 63999
Number of extents after reconfiguration: 63999
Physical volume “/dev/disk/disk954” was changed.

Volume Group configuration for /dev/vgLP0data has been saved.

Add the disk back and it will show the correct Total PE and Free PE

root@hpux:~ # vgextend -g pvgLP0data vgLP0data /dev/disk/disk670
Volume group “vgLP0data” has been successfully extended.
Physical volume group “pvgLP0data” has been successfully extended.
Volume Group configuration for /dev/vgLP0data has been saved in /etc/lvmconf/vgLP0data.conf

root@hpux:~ # vgdisplay -v vgLP0data
— Volume groups —
VG Name /dev/vgLP0data
VG Write Access read/write
VG Status available, exclusive
Max LV 2047
Cur LV 8
Open LV 8
Cur Snapshot LV 0
Max PV 2048
Cur PV 8
Act PV 8
Max PE per PV 524288
VGDA 16
PE Size (Mbytes) 32
Unshare unit size (Kbytes) 1024
Total PE 511992
Alloc PE 447986
Current pre-allocated PE 0
Free PE 64006
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 2.2
VG Max Size 20t
VG Max Extents 655360
Cur Snapshot Capacity 0p
Max Snapshot Capacity 20t

— Logical volumes —
LV Name /dev/vgLP0data/lvsapdata1
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata2
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata3
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata4
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata5
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata6
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata7
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata8
LV Status available/syncd
LV Size (Mbytes) 0
Current LE 0
Allocated PE 0
Used PV 0

— Physical volumes —
PV Name /dev/disk/disk949
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk950
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk951
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk952
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk953
PV Status available
Total PE 63999
Free PE 2
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk954
PV Status available
Total PE 63999
Free PE 2
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk955
PV Status available
Total PE 63999
Free PE 3
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk670
PV Status available
Total PE 63999
Free PE 63999
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

— Physical volume groups —
PVG Name pvgLP0data
PV Name /dev/disk/disk949
PV Name /dev/disk/disk950
PV Name /dev/disk/disk951
PV Name /dev/disk/disk952
PV Name /dev/disk/disk953
PV Name /dev/disk/disk954
PV Name /dev/disk/disk955
PV Name /dev/disk/disk670

HP-UX: Adding a new disk that is part of a Physical Volume Group (PVG)

root@hpux:~ # vgdisplay -v vgLP0data
— Volume groups —
VG Name /dev/vgLP0data
VG Write Access read/write
VG Status available, exclusive
Max LV 2047
Cur LV 7
Open LV 7
Cur Snapshot LV 0
Max PV 2048
Cur PV 7
Act PV 7
Max PE per PV 447994
VGDA 14
PE Size (Mbytes) 32
Unshare unit size (Kbytes) 1024
Total PE 447993
Alloc PE 447986
Current pre-allocated PE 0
Free PE 7
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 2.2
VG Max Size 14335819m
VG Max Extents 447994
Cur Snapshot Capacity 0p
Max Snapshot Capacity 14335819m

— Logical volumes —
LV Name /dev/vgLP0data/lvsapdata1
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata2
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata3
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata4
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata5
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata6
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

LV Name /dev/vgLP0data/lvsapdata7
LV Status available/syncd
LV Size (Mbytes) 2047936
Current LE 63998
Allocated PE 63998
Used PV 7

— Physical volumes —
PV Name /dev/disk/disk949
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk950
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk951
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk952
PV Status available
Total PE 63999
Free PE 0
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk953
PV Status available
Total PE 63999
Free PE 2
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk954
PV Status available
Total PE 63999
Free PE 2
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk955
PV Status available
Total PE 63999
Free PE 3
Current pre-allocated PE 0
Autoswitch On
Proactive Polling On

— Physical volume groups —
PVG Name pvgLP0data
PV Name /dev/disk/disk949
PV Name /dev/disk/disk950
PV Name /dev/disk/disk951
PV Name /dev/disk/disk952
PV Name /dev/disk/disk953
PV Name /dev/disk/disk954
PV Name /dev/disk/disk955

Add the disk to LVM

root@hpux:~ # pvcreate /dev/rdisk/disk670
Physical volume “/dev/rdisk/disk670” has been successfully created.

Add the disk to volume group and add the flag -g to add to PVG section

root@hpux:~ # vgextend -g pvgLP0data vgLP0data /dev/disk/disk670
Volume group “vgLP0data” has been successfully extended.
Physical volume group “pvgLP0data” has been successfully extended.
Volume Group configuration for /dev/vgLP0data has been saved in /etc/lvmconf/vgLP0data.conf

Clustered Linux server showing device-mapper: multipath: Failing path in /var/log/messages

I have a disk presented to 4 servers.

Everyday we receive a notification saying that a specific multipathed disk lost all paths.

The disk is showing a SCSI reservation conflict – SCSI persistent reservations provide the capability to control the access of each node to shared storage devices

May 11 13:35:04 linux kernel: sd 0:0:0:38: reservation conflict
May 11 13:35:04 linux kernel: sd 0:0:0:38: [sdag] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 0:0:0:38: [sdag] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00

May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdag, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 66:0. <————————sdag
May 11 13:35:04 linux kernel: sd 1:0:0:38: reservation conflict
May 11 13:35:04 linux kernel: sd 1:0:0:38: [sdeh] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 1:0:0:38: [sdeh] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdeh, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 128:144. <———————-sdeh
May 11 13:35:04 linux multipathd: 66:0: mark as failed
May 11 13:35:04 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 3

May 11 13:35:04 linux kernel: sd 0:0:1:38: reservation conflict
May 11 13:35:04 linux kernel: sd 0:0:1:38: [sdcc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 0:0:1:38: [sdcc] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdcc, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 69:0.<—————————sdcc
May 11 13:35:04 linux kernel: sd 1:0:1:38: reservation conflict
May 11 13:35:04 linux kernel: sd 1:0:1:38: [sdgg] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
May 11 13:35:04 linux kernel: sd 1:0:1:38: [sdgg] CDB: Write(10): 2a 00 00 00 14 50 00 00 08 00
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev sdgg, sector 5200
May 11 13:35:04 linux kernel: device-mapper: multipath: Failing path 131:192.<————————-sdgg
May 11 13:35:04 linux kernel: end_request: critical nexus error, dev dm-209, sector 5200

May 11 13:35:05 linux multipathd: 128:144: mark as failed
May 11 13:35:05 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 2
May 11 13:35:05 linux multipathd: 69:0: mark as failed
May 11 13:35:05 linux multipathd: PP0_oraarch_disk_001: remaining active paths: 1
May 11 13:35:05 linux multipathd: 131:192: mark as failed

PP0_oraarch_disk_001 (350002ad05071374b) dm-209 3PARdata,VV
size=300G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
|- 0:0:0:38 sdag 66:0 active undef running
|- 1:0:0:38 sdeh 128:144 active undef running
|- 0:0:1:38 sdcc 69:0 active undef running
`- 1:0:1:38 sdgg 131:192 active undef running

Since this is opening support tickets, I removed monitoring for this disk

lvcreate – Aborting. Failed to wipe start of new LV

root@linux:~ # lvcreate -L 400g -n lv_test vg_test
Volume “vg_test/lv_test” is not active locally.
Aborting. Failed to wipe start of new LV.

Add vg_test to volume_list parameter in /etc/lvm/lvm.conf

vi /etc/lvm/lvm.conf
volume_list = [ “vgo2o”, “vgroot”, “vg_test” ]

Then try again

root@linux:~ # lvcreate -L 400g -n lv_test vg_test
Logical volume “lv_test” created.

Rotate file /var/log/faillog, /var/log/lastlog and /var/log/tallylog

If you are having disk space problems in /var and you found that /var/log/faillog, /var/log/lastlog and /var/log/tallylog are filling up the space and you need to rotate them, you probably don’t need to rotate them

root@linux:~ # df -h /var/log
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgroot-lv_var_log
4.9G 2.0G 2.8G 42% /var/log

These are sparse files and are occupying minimal disk space. Keep looking for the offender

root@linux:~ # ls -lh /var/log/faillog
-rw——-. 1 root root 258M Apr 25 14:07 /var/log/faillog

root@linux:~ # du -sh /var/log/faillog
624K /var/log/faillog
root@linux:~ # du -h –apparent-size /var/log/faillog
258M /var/log/faillog

root@linux:~ # ls -lh /var/log/lastlog
-rw——-. 1 root root 2.3G May 2 11:05 /var/log/lastlog

root@linux:~ # du -h /var/log/lastlog
348K /var/log/lastlog
root@linux:~ # du -h –apparent-size /var/log/lastlog
2.3G /var/log/lastlog

root@linux:~ # ls -lh /var/log/tallylog
-rw——-. 1 root root 515M May 2 10:42 /var/log/tallylog

root@linux:~ # du -sh /var/log/tallylog
288K /var/log/tallylog
root@linux:~ # du -sh –apparent-size /var/log/tallylog
515M /var/log/tallylog

Source: faillog command create a huge file, like 128GB file (/var/log/faillog)

Why is the /var/log/lastlog file so large?

Failure: At least one ITO agent process is not running on linux.setaoffice.com

I was having some problems with opcacta

root@linux:~ # ovc -status
coda OV Performance Core COREXT (23107) Running
opcacta OVO Action Agent AGENT,EA Aborted
opcle OVO Logfile Encapsulator AGENT,EA (23073) Running
opcmona OVO Monitor Agent AGENT,EA (6052) Running
opcmsga OVO Message Agent AGENT,EA (22896) Running
opcmsgi OVO Message Interceptor AGENT,EA (22979) Running
ovbbccb OV Communication Broker CORE (22700) Running
ovcd OV Control CORE (22687) Running
ovconfd OV Config and Deploy COREXT (22725) Running

Looking at /var/opt/OV/log/System.txt after stopping and starting, it shows a message saying that there is a problem with opcacta

root@linux:~ # tail -20 /var/opt/OV/log/System.txt
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:31 2016: opcacta (10806/47113196044032): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:31 2016: ovcd (10437/47099555100992): (ctrl-208) Component ‘opcacta’ with pid 10806 exited with exit value ‘0’. Restarting component.
0: ERR: Fri Apr 29 10:47:36 2016: ovcd (10437/47099477543232): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:36 2016: ovc (10434/47239787368608): (ctrl-7) Error in the target component.
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:36 2016: ovcd (10437/47099573193024): (ctrl-208) Component ‘opcacta’ with pid 12760 exited with exit value ‘0’. Restarting component.
0: INF: Fri Apr 29 10:47:37 2016: coda (10779/47049355707824): SCOPE datasource initialization succeeded
0: ERR: Fri Apr 29 10:47:41 2016: ovcd (10437/47099450292544): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: ERR: Fri Apr 29 10:47:42 2016: ovcd (10437/47099573193024): (ctrl-94) Component ‘opcacta’ exited after very short runtime, therefore it will not be automatically restarted. Use ‘ovc -start opcacta’.

Stop OVC

root@linux:~ # ovc -kill

Stop OVPA

root@linux:~ # ovpa stop

Shutting down Perf Agent collection software
Shutting down scopeux, pid(s) 13146
Waiting on 13146 (10 more tries)
The Perf Agent collector, scopeux has been shut down successfully.
NOTE: The ARM registration daemon ttd will be left running.

Shutting down the alarm generator perfalarm, pid(s) 13244
The perfalarm process has terminated

Remove log* files under /var/opt/perf/datafiles/

root@linux:~ # rm /var/opt/perf/datafiles/log*
rm: remove regular file `/var/opt/perf/datafiles/logappl’? y
rm: remove regular file `/var/opt/perf/datafiles/logdev’? y
rm: remove regular file `/var/opt/perf/datafiles/logglob’? y
rm: remove regular file `/var/opt/perf/datafiles/logindx’? y
rm: remove regular file `/var/opt/perf/datafiles/logpcmd0′? y
rm: remove regular file `/var/opt/perf/datafiles/logproc’? y
rm: remove regular file `/var/opt/perf/datafiles/logtran’? y

Remove *q files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*q
rm: remove regular file `/var/opt/OV/tmp/OpC/aaoorbCvq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/actagtq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpicmaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpimaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtq’? y

Remove *df files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*df
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtdf’? y

Restart HPOM and OVPA

root@linux:~ # /opt/perf/bin/ovpa start

The Perf Agent scope collector is being started.
The ARM registration daemon ttd is already running.
It will be signaled to reprocess its configuration file.

The Performance collection daemon
/opt/perf/bin/scopeux has been started.

The coda daemon /opt/OV/lbin/perf/coda has been started.
It will be fully operational in a few minutes.

The Perf Agent alarm generator is being started.
The alarm generator /opt/perf/bin/perfalarm
has been started.

root@linux:~ # ovc -start

root@linux:~ # ovc -status
coda OV Performance Core COREXT (22782) Running
opcacta OVO Action Agent AGENT,EA (23343) Running
opcle OVO Logfile Encapsulator AGENT,EA (23426) Running
opcmona OVO Monitor Agent AGENT,EA (23501) Running
opcmsga OVO Message Agent AGENT,EA (23238) Running
opcmsgi OVO Message Interceptor AGENT,EA (23459) Running
ovbbccb OV Communication Broker CORE (22755) Running
ovcd OV Control CORE (22747) Running
ovconfd OV Config and Deploy COREXT (22889) Running

Follow

Get every new post delivered to your Inbox.

Join 3,013 other followers

%d bloggers like this: