Tag: dmesg

UXMON: EXT3-fs: checktime reached, running e2fsck is recommended

Node : linux.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : warning
OM Server Time: 2016-08-31 10:22:49
Message : UXMON: EXT3-fs: checktime reached, running e2fsck is recommended
Msg Group : OS
Application : dmsg_mon
Object : EXT3
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Check file /etc/fstab to see if your root filesystem has the number 1 . All the other filesystems should have a number 2

root@linux:~ # cat /etc/fstab
/dev/cciss/c0d0p1 / ext3 acl,user_xattr 1 1
/dev/cciss/c0d0p3 /boot ext3 acl,user_xattr 1 2
/dev/localVG/var /var ext3 acl,user_xattr 1 2
/dev/localVG/app.software /usr/software ext3 acl,user_xattr 1 2
/dev/localVG/dat.users /usr/users ext3 acl,user_xattr 1 2
/dev/localVG/dat.audit /audit ext3 acl,user_xattr 1 2
/dev/cciss/c0d0p2 swap swap defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
usbfs /proc/bus/usb usbfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0

Create the file /forcefsck and reboot the server if possible

root@linux:~ # ls -l /forcefsck
/bin/ls: /forcefsck: No such file or directory

root@linux:~ # touch /forcefsck

root@linux:~ # shutdown -r now

Run dmesg -c twice to clear the dmesg buffer

root@linux:~ # dmesg -c
root@linux:~ # dmesg -c

DMESG-UNCLASSIFIED: ia64dsk: The disk for dev_t bc4e0100 appears to have grown since the partition t

Node : hpux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : major
OM Server Time: 2016-01-02 21:07:51
Message : DMESG-UNCLASSIFIED: ia64dsk: The disk for dev_t bc4e0100 appears to have grown since the partition table was written.
Msg Group : OS
Application : HPUX_dmesg
Object : dmesg_UNCLASSIFIED
Event Type :
not_found

Instance Name :
not_found

Instruction : No

Checking which disks are alarming

root@hpux:~ # dmesg | grep ia64dsk
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc420100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t bc430100 appears to have grown since the partition table was written.

Checking system device

root@hpux:~ # ls -ltraR /dev | grep 420100
crw-r—– 1 bin sys 188 0x420100 Feb 12 2015 c66t0d1
brw-r—– 1 bin sys 31 0x420100 Feb 12 2015 c66t0d1

Checking hardware path

root@hpux:~ # ioscan -kfnC disk /dev/dsk/c66t0d1
Class I H/W Path Driver S/W State H/W Type Description
==================================================================
disk 48 0/2/1/1.10.234.64.0.0.1 sdisk CLAIMED DEVICE 3PARdataVV
/dev/dsk/c66t0d1 /dev/dsk/c66t0d1s2 /dev/rdsk/c66t0d1 /dev/rdsk/c66t0d1s2
/dev/dsk/c66t0d1s1 /dev/dsk/c66t0d1s3 /dev/rdsk/c66t0d1s1 /dev/rdsk/c66t0d1s3

root@hpux:~ # ioscan -m hwpath -H 0/2/1/1.10.234.64.0.0.1
Lun H/W Path Lunpath H/W Path Legacy H/W Path
====================================================================
64000/0xfa00/0x9a
0/2/1/1.0x21230002ac001673.0x4001000000000000 0/2/1/1.10.218.65.0.0.1
0/2/1/1.10.234.64.0.0.1

root@hpux:~ # scsimgr -f replace_wwid -H 64000/0xfa00/0x9a
scsimgr: Successfully validated binding of LUN paths with new LUN.

root@hpux:~ # ls -ltraR /dev | grep 430100
crw-r—– 1 bin sys 188 0x430100 Feb 12 2015 c67t0d1
brw-r—– 1 bin sys 31 0x430100 Feb 12 2015 c67t0d1

root@hpux:~ # ioscan -kfnC disk /dev/dsk/c67t0d1
Class I H/W Path Driver S/W State H/W Type Description
==================================================================
disk 59 0/5/1/0.20.218.64.0.0.1 sdisk CLAIMED DEVICE 3PARdataVV
/dev/dsk/c67t0d1 /dev/dsk/c67t0d1s2 /dev/rdsk/c67t0d1 /dev/rdsk/c67t0d1s2
/dev/dsk/c67t0d1s1 /dev/dsk/c67t0d1s3 /dev/rdsk/c67t0d1s1 /dev/rdsk/c67t0d1s3

root@hpux:~ # ioscan -m hwpath -H 0/5/1/0.20.218.64.0.0.1
Lun H/W Path Lunpath H/W Path Legacy H/W Path
====================================================================
64000/0xfa00/0x9a
0/5/1/0.0x20240002ac001673.0x4001000000000000 0/5/1/0.20.218.64.0.0.1

root@hpux:~ # scsimgr -f replace_wwid -H 64000/0xfa00/0x9a
scsimgr: Successfully validated binding of LUN paths with new LUN.

Replacing root disk / root mirror for Solaris server with Solaris Volume Manager

Reviewing metadevice state database

root@solaris10:/ # metadb
flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t1d0s7
a p luo 8208 8192 /dev/dsk/c0t1d0s7
a p luo 16400 8192 /dev/dsk/c0t1d0s7
a p luo 16 8192 /dev/dsk/c0t0d0s7
a p luo 8208 8192 /dev/dsk/c0t0d0s7
a p luo 16400 8192 /dev/dsk/c0t0d0s7

The disk to replace is the c0t0d0

Deleting metadevice state database from the disk that is being replaced

root@solaris10:/ # metadb -d /dev/dsk/c0t0d0s7

Reviewing metadevices and disk slices

root@solaris10:/ # metastat -c
d40 m 516MB d41 d42
d41 s 516MB c0t0d0s6
d42 s 516MB c0t1d0s6
d20 m 20GB d21 d22
d21 s 20GB c0t0d0s1
d22 s 20GB c0t1d0s1
d30 m 23GB d31 d32
d31 s 23GB c0t0d0s5
d32 s 23GB c0t1d0s5
d10 m 24GB d11 d12
d11 s 24GB c0t0d0s0
d12 s 24GB c0t1d0s0

Splitting mirror

root@solaris10:/ # metadetach d10 d11
d10: submirror d11 is detached
root@solaris10:/ # metadetach d20 d21
d20: submirror d21 is detached
root@solaris10:/ # metadetach d30 d31
d30: submirror d31 is detached
root@solaris10:/ # metadetach d40 d41
d40: submirror d41 is detached

Removing the metadevice

root@solaris10:/ # metaclear d11
d11: Concat/Stripe is cleared
root@solaris10:/ # metaclear d21
d21: Concat/Stripe is cleared
root@solaris10:/ # metaclear d31
d31: Concat/Stripe is cleared
root@solaris10:/ # metaclear d41
d41: Concat/Stripe is cleared

Identifying the disk

root@solaris10:/ # format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
2. c6t60060E8004587800000058780000012Cd0
/scsi_vhci/ssd@g60060e8004587800000058780000012c
3. c6t60060E8004587800000058780000010Ed0 <HITACHI-OPEN-V*4-SUN-5009 cyl 48744 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000010e
4. c6t60060E8004587800000058780000012Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000012e
5. c6t60060E8004587800000058780000011Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000011e
6. c6t60060E8004587800000058780000011Fd0 <HITACHI-OPEN-V*2-SUN-5009 cyl 24371 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000011f
7. c6t60060E80045878000000587800000088d0
/scsi_vhci/ssd@g60060e80045878000000587800000088
8. c6t60060E80045878000000587800000087d0
/scsi_vhci/ssd@g60060e80045878000000587800000087
9. c6t60060E80045878000000587800000084d0
/scsi_vhci/ssd@g60060e80045878000000587800000084
10. c6t60060E80045878000000587800000082d0
/scsi_vhci/ssd@g60060e80045878000000587800000082
11. c6t60060E80045878000000587800000080d0
/scsi_vhci/ssd@g60060e80045878000000587800000080
12. c6t60060E80045878000000587800000013d0
/scsi_vhci/ssd@g60060e80045878000000587800000013
13. c6t60060E80045878000000587800000012d0
/scsi_vhci/ssd@g60060e80045878000000587800000012
14. c6t60060E80045878000000587800000089d0
/scsi_vhci/ssd@g60060e80045878000000587800000089
Specify disk (enter its number):

After selecting the disk, select analyze

Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit
format>

Perform a read only test

format> analyze

ANALYZE MENU:
read – read only test (doesn’t harm SunOS)
refresh – read then write (doesn’t harm data)
test – pattern testing (doesn’t harm data)
write – write then read (corrupts data)
compare – write, read, compare (corrupts data)
purge – write, read, write (corrupts data)
verify – write entire disk, then verify (corrupts data)
print – display data buffer
setup – set analysis parameters
config – show analysis parameters
! – execute , then return
quit
analyze>

Type read and go check which disk have the LED blinking. Before unplugging the disk, interrupt the test with CTRL+C

analyze> read
Ready to analyze (won’t harm SunOS). This takes a long time,
but is interruptable with CTRL-C. Continue? y

pass 0
^C 818/4/164 CTRL+C
Total of 0 defective blocks repaired.
analyze>

Quit format

analyze> quit

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit
format> quit

dmesg shows that the disk has been replaced

root@solaris10:/ # dmesg
Nov 24 05:00:01 solaris10 xntpd[10967]: [ID 266339 daemon.notice] using kernel phase-lock loop 0041, drift correction -239.18301
Nov 24 10:52:18 solaris10 scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0 (mpt0):
Nov 24 10:52:18 solaris10 scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0 (mpt0):
Nov 24 10:52:18 solaris10 mpt_handle_event_sync : SAS target 0 not responding.
Nov 24 10:52:18 solaris10 mpt_handle_event_sync : SAS target 0 not responding.
Nov 24 10:52:23 solaris10 SC Alert: [ID 209909 daemon.error] DISK at HDD0 has been removed.
Nov 24 10:52:23 solaris10 SC Alert: [ID 209909 daemon.error] DISK at HDD0 has been removed.
Nov 24 10:52:39 solaris10 scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0 (mpt0):
Nov 24 10:52:39 solaris10 scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0 (mpt0):
Nov 24 10:52:39 solaris10 mpt_handle_event_sync : SAS target 0 added.
Nov 24 10:52:39 solaris10 mpt_handle_event_sync : SAS target 0 added.
Nov 24 10:52:39 solaris10 scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0 (mpt0):
Nov 24 10:52:39 solaris10 scsi: [ID 107833 kern.warning] WARNING: /pci@780/pci@0/pci@9/scsi@0 (mpt0):
Nov 24 10:52:39 solaris10 wwn for target has changed
Nov 24 10:52:39 solaris10 wwn for target has changed
Nov 24 10:52:41 solaris10 SC Alert: [ID 735619 daemon.error] DISK at HDD0 has been inserted.
Nov 24 10:52:41 solaris10 SC Alert: [ID 735619 daemon.error] DISK at HDD0 has been inserted.

Force Solaris to rescan the disk. You can skip these steps

root@solaris10:/ # format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
2. c6t60060E8004587800000058780000012Cd0
/scsi_vhci/ssd@g60060e8004587800000058780000012c
3. c6t60060E8004587800000058780000010Ed0 <HITACHI-OPEN-V*4-SUN-5009 cyl 48744 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000010e
4. c6t60060E8004587800000058780000012Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000012e
5. c6t60060E8004587800000058780000011Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000011e
6. c6t60060E8004587800000058780000011Fd0 <HITACHI-OPEN-V*2-SUN-5009 cyl 24371 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000011f
7. c6t60060E80045878000000587800000088d0
/scsi_vhci/ssd@g60060e80045878000000587800000088
8. c6t60060E80045878000000587800000087d0
/scsi_vhci/ssd@g60060e80045878000000587800000087
9. c6t60060E80045878000000587800000084d0
/scsi_vhci/ssd@g60060e80045878000000587800000084
10. c6t60060E80045878000000587800000082d0
/scsi_vhci/ssd@g60060e80045878000000587800000082
11. c6t60060E80045878000000587800000080d0
/scsi_vhci/ssd@g60060e80045878000000587800000080
12. c6t60060E80045878000000587800000013d0
/scsi_vhci/ssd@g60060e80045878000000587800000013
13. c6t60060E80045878000000587800000012d0
/scsi_vhci/ssd@g60060e80045878000000587800000012
14. c6t60060E80045878000000587800000089d0
/scsi_vhci/ssd@g60060e80045878000000587800000089
Specify disk (enter its number):

Select the disk 0 – c0t0d0

Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit
format>

Select type and Auto configure

format> t

AVAILABLE DRIVE TYPES:
0. Auto configure
1. Quantum ProDrive 80S
2. Quantum ProDrive 105S
3. CDC Wren IV 94171-344
4. SUN0104
5. SUN0207
6. SUN0327
7. SUN0340
8. SUN0424
9. SUN0535
10. SUN0669
11. SUN1.0G
12. SUN1.05
13. SUN1.3G
14. SUN2.1G
15. SUN2.9G
16. Zip 100
17. Zip 250
18. Peerless 10GB
19. SUN72G
20. FUJITSU-MAY2073RCSUN72G-0501
21. HITACHI-OPEN-V-SUN-5009
22. HITACHI-OPEN-V*2-SUN-5009
23. HITACHI-OPEN-V*4-SUN-5009
24. other
Specify disk type (enter its number)[19]: 0
c0t0d0: configured with capacity of 68.35GB

selecting c0t0d0
[disk formatted]
format> label
Ready to label disk, continue? y

format> quit

Verifying disk partition

root@solaris10:/ # format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
2. c6t60060E8004587800000058780000012Cd0
/scsi_vhci/ssd@g60060e8004587800000058780000012c
3. c6t60060E8004587800000058780000010Ed0 <HITACHI-OPEN-V*4-SUN-5009 cyl 48744 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000010e
4. c6t60060E8004587800000058780000012Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000012e
5. c6t60060E8004587800000058780000011Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000011e
6. c6t60060E8004587800000058780000011Fd0 <HITACHI-OPEN-V*2-SUN-5009 cyl 24371 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000011f
7. c6t60060E80045878000000587800000088d0
/scsi_vhci/ssd@g60060e80045878000000587800000088
8. c6t60060E80045878000000587800000087d0
/scsi_vhci/ssd@g60060e80045878000000587800000087
9. c6t60060E80045878000000587800000084d0
/scsi_vhci/ssd@g60060e80045878000000587800000084
10. c6t60060E80045878000000587800000082d0
/scsi_vhci/ssd@g60060e80045878000000587800000082
11. c6t60060E80045878000000587800000080d0
/scsi_vhci/ssd@g60060e80045878000000587800000080
12. c6t60060E80045878000000587800000013d0
/scsi_vhci/ssd@g60060e80045878000000587800000013
13. c6t60060E80045878000000587800000012d0
/scsi_vhci/ssd@g60060e80045878000000587800000012
14. c6t60060E80045878000000587800000089d0
/scsi_vhci/ssd@g60060e80045878000000587800000089
Specify disk (enter its number):

Selecting disk c0t0d0

Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit
format> p

Printing disk partitions

PARTITION MENU:
0 – change `0′ partition
1 – change `1′ partition
2 – change `2′ partition
3 – change `3′ partition
4 – change `4′ partition
5 – change `5′ partition
6 – change `6′ partition
7 – change `7′ partition
select – select a predefined table
modify – modify a predefined partition table
name – name the current table
print – display the current table
label – write partition map and label to the disk
! – execute , then return
quit
partition> p

The most important thing to check is disk slice 2 – backup

Current partition table (original):
Total disk cylinders available: 14087 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 – 25 129.19MB (26/0/0) 264576
1 swap wu 26 – 51 129.19MB (26/0/0) 264576
2 backup wu 0 – 14086 68.35GB (14087/0/0) 143349312
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 usr wm 52 – 14086 68.10GB (14035/0/0) 142820160
7 unassigned wm 0 0 (0/0/0) 0

Check the other mirror side – c0t1d0

format> disk

AVAILABLE DISK SELECTIONS:
0. c0t0d0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
2. c6t60060E8004587800000058780000012Cd0
/scsi_vhci/ssd@g60060e8004587800000058780000012c
3. c6t60060E8004587800000058780000010Ed0 <HITACHI-OPEN-V*4-SUN-5009 cyl 48744 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000010e
4. c6t60060E8004587800000058780000012Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000012e
5. c6t60060E8004587800000058780000011Ed0
/scsi_vhci/ssd@g60060e8004587800000058780000011e
6. c6t60060E8004587800000058780000011Fd0 <HITACHI-OPEN-V*2-SUN-5009 cyl 24371 alt 2 hd 15 sec 512>
/scsi_vhci/ssd@g60060e8004587800000058780000011f
7. c6t60060E80045878000000587800000088d0
/scsi_vhci/ssd@g60060e80045878000000587800000088
8. c6t60060E80045878000000587800000087d0
/scsi_vhci/ssd@g60060e80045878000000587800000087
9. c6t60060E80045878000000587800000084d0
/scsi_vhci/ssd@g60060e80045878000000587800000084
10. c6t60060E80045878000000587800000082d0
/scsi_vhci/ssd@g60060e80045878000000587800000082
11. c6t60060E80045878000000587800000080d0
/scsi_vhci/ssd@g60060e80045878000000587800000080
12. c6t60060E80045878000000587800000013d0
/scsi_vhci/ssd@g60060e80045878000000587800000013
13. c6t60060E80045878000000587800000012d0
/scsi_vhci/ssd@g60060e80045878000000587800000012
14. c6t60060E80045878000000587800000089d0
/scsi_vhci/ssd@g60060e80045878000000587800000089
Specify disk (enter its number)[0]: 1

Check disk partition

selecting c0t1d0
[disk formatted]
format> p

PARTITION MENU:
0 – change `0′ partition
1 – change `1′ partition
2 – change `2′ partition
3 – change `3′ partition
4 – change `4′ partition
5 – change `5′ partition
6 – change `6′ partition
7 – change `7′ partition
select – select a predefined table
modify – modify a predefined partition table
name – name the current table
print – display the current table
label – write partition map and label to the disk
! – execute , then return
quit
partition> p
Current partition table (original):
Total disk cylinders available: 14087 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 – 5031 24.42GB (5032/0/0) 51205632
1 swap wu 5032 – 9153 20.00GB (4122/0/0) 41945472
2 backup wm 0 – 14086 68.35GB (14087/0/0) 143349312
3 unassigned wu 0 0 (0/0/0) 0
4 unassigned wu 0 0 (0/0/0) 0
5 var wm 9154 – 13951 23.28GB (4798/0/0) 48824448
6 unassigned wm 13952 – 14055 516.75MB (104/0/0) 1058304
7 unassigned wm 14056 – 14086 154.03MB (31/0/0) 315456

partition>

Compare partition table for the two disks. Both have the same block number
====================================================================================================

partition> p
Current partition table (original):
Total disk cylinders available: 14087 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 – 25 129.19MB (26/0/0) 264576
1 swap wu 26 – 51 129.19MB (26/0/0) 264576
2 backup wu 0 – 14086 68.35GB ( 14087/0/0) 143349312
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 usr wm 52 – 14086 68.10GB (14035/0/0) 142820160
7 unassigned wm 0 0 (0/0/0) 0

partition> p
Current partition table (original):
Total disk cylinders available: 14087 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 – 5031 24.42GB (5032/0/0) 51205632
1 swap wu 5032 – 9153 20.00GB (4122/0/0) 41945472
2 backup wm 0 – 14086 68.35GB ( 14087/0/0) 143349312
3 unassigned wu 0 0 (0/0/0) 0
4 unassigned wu 0 0 (0/0/0) 0
5 var wm 9154 – 13951 23.28GB (4798/0/0) 48824448
6 unassigned wm 13952 – 14055 516.75MB (104/0/0) 1058304
7 unassigned wm 14056 – 14086 154.03MB (31/0/0) 315456

====================================================================================================

Even if the brand is different, same number of blocks, proceeding

0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c0t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@1,0

Copying disk partitioning to the newly inserted disk

root@solaris10:/ # prtvtoc /dev/rdsk/c0t1d0s2 | fmthard -s – /dev/rdsk/c0t0d0s2
fmthard: New volume table of contents now in place.

Now c0t0d0 have the same partitions as c0t1d0

partition> p
Current partition table (original):
Total disk cylinders available: 14087 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 – 5031 24.42GB (5032/0/0) 51205632
1 swap wu 5032 – 9153 20.00GB (4122/0/0) 41945472
2 backup wm 0 – 14086 68.35GB (14087/0/0) 143349312
3 unassigned wu 0 0 (0/0/0) 0
4 unassigned wu 0 0 (0/0/0) 0
5 var wm 9154 – 13951 23.28GB (4798/0/0) 48824448
6 unassigned wm 13952 – 14055 516.75MB (104/0/0) 1058304
7 unassigned wm 14056 – 14086 154.03MB (31/0/0) 315456

Recreating metadevice state database

root@solaris10:/ # metadb -a -f -c3 c0t0d0s7
root@solaris10:/ # metadb
flags first blk block count
a m p luo 16 8192 /dev/dsk/c0t1d0s7
a p luo 8208 8192 /dev/dsk/c0t1d0s7
a p luo 16400 8192 /dev/dsk/c0t1d0s7
a u 16 8192 /dev/dsk/c0t0d0s7
a u 8208 8192 /dev/dsk/c0t0d0s7
a u 16400 8192 /dev/dsk/c0t0d0s7

Recreating metadevices

root@solaris10:/ # metainit d11 1 1 c0t0d0s0
d11: Concat/Stripe is setup
root@solaris10:/ # metainit d21 1 1 c0t0d0s1
d21: Concat/Stripe is setup
root@solaris10:/ # metainit d31 1 1 c0t0d0s5
d31: Concat/Stripe is setup
root@solaris10:/ # metainit d41 1 1 c0t0d0s6
d41: Concat/Stripe is setup

Reattaching metadevices

root@solaris10:/ # metattach d10 d11
d10: submirror d11 is attached
root@solaris10:/ # metattach d20 d21
d20: submirror d21 is attached
root@solaris10:/ # metattach d30 d31
d30: submirror d31 is attached
root@solaris10:/ # metattach d40 d41
d40: submirror d41 is attached

Checking status

root@solaris10:/ # metastat -c
d40 m 516MB d42 d41 (resync-1%)
d42 s 516MB c0t1d0s6
d41 s 516MB c0t0d0s6
d20 m 20GB d22 d21 (resync-0%)
d22 s 20GB c0t1d0s1
d21 s 20GB c0t0d0s1
d30 m 23GB d32 d31 (resync-0%)
d32 s 23GB c0t1d0s5
d31 s 23GB c0t0d0s5
d10 m 24GB d12 d11 (resync-0%)
d12 s 24GB c0t1d0s0
d11 s 24GB c0t0d0s0

DMESG-UNCLASSIFIED: to probe existing LUN id 0x0 failed with errno of 6.

DMESG-UNCLASSIFIED: to probe existing LUN id 0x0 failed with errno of 6.

Node : hp-ux.setaoffice.com
Node Type : Itanium 64/32(HTTPS)
Severity : major
OM Server Time: 2015-11-06 10:30:53
Message : DMESG-UNCLASSIFIED: to probe existing LUN id 0x0 failed with errno of 6.
Msg Group : OS
Application : HPUX_dmesg
Object : dmesg_UNCLASSIFIED
Event Type :
not_found

Instance Name :
not_found

Instruction : No

 

Message showing in dmesg and /var/adm/syslog/syslog.log

root@hp-ux:/root # dmesg | grep probe
An attempt to probe existing LUN id 0x0 failed with errno of 22.
An attempt to probe existing LUN id 0x0 failed with errno of 22.

root@hp-ux:/root # grep probe /var/adm/syslog/syslog.log | tail
Nov 4 16:08:31 vlunx012 vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Nov 4 16:08:31 vlunx012 vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 6.
Nov 4 16:08:33 vlunx012 vmunix: An attempt to probe existing LUN id 0x0 failed with errno of 22.

Scanning the hardware and it is shown that some tapes are in NO_HW status. Removed these tapes

root@hp-ux:/root # ioscan -fnkNC lunpath | grep NO_HW
lunpath 139 0/0/4/0.0xb.0x0 eslpt NO_HW LUN_PATH LUN path for tape95
lunpath 147 0/0/4/0.0×19.0x0 eslpt NO_HW LUN_PATH LUN path for tape99
lunpath 141 0/0/4/0.0x1a.0x0 eslpt NO_HW LUN_PATH LUN path for tape96
lunpath 144 0/0/4/0.0x1c.0x0 eslpt NO_HW LUN_PATH LUN path for tape97
lunpath 155 0/0/4/0.0x1d.0x0 eslpt NO_HW LUN_PATH LUN path for tape115
lunpath 150 0/0/4/0.0x1e.0x0 eslpt NO_HW LUN_PATH LUN path for tape47
lunpath 157 0/0/4/0.0×24.0x0 eslpt NO_HW LUN_PATH LUN path for tape98
lunpath 408 0/0/6/0.0×26.0x0 eslpt NO_HW LUN_PATH LUN path for tape199
lunpath 410 0/0/6/0.0×27.0x0 eslpt NO_HW LUN_PATH LUN path for tape203
lunpath 412 0/0/6/0.0×28.0x0 eslpt NO_HW LUN_PATH LUN path for tape200
lunpath 415 0/0/6/0.0×29.0x0 eslpt NO_HW LUN_PATH LUN path for tape201
lunpath 418 0/0/6/0.0x2a.0x0 eslpt NO_HW LUN_PATH LUN path for tape212
lunpath 423 0/0/6/0.0x2b.0x0 eslpt NO_HW LUN_PATH LUN path for tape204

root@hp-ux:/root # rmsf -H 0/0/4/0.0xb.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0×19.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1a.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1c.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1d.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0x1e.0x0
root@hp-ux:/root # rmsf -H 0/0/4/0.0×24.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×26.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×27.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×28.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0×29.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0x2a.0x0
root@hp-ux:/root # rmsf -H 0/0/6/0.0x2b.0x0

root@hp-ux:/root # ioscan -fnkNC lunpath | grep NO_HW
root@hp-ux:/root #

dmesg output
class : lunpath, instance 139
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 147
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 141
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 144
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 155
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 150
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 157
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 408
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 410
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 412
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 415
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 418
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 423
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 139
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 147
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 141
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 144
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 155
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 150
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 157
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 412
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 408
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 410
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 415
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 418
An attempt to probe existing LUN id 0x0 failed with errno of 22.

class : lunpath, instance 423
An attempt to probe existing LUN id 0x0 failed with errno of 22.

HPOM – SOL_mon is warning about a network interface

UXMON: genunix: [ID 408789 kern.warning] WARNING: ce1: fault detected external to device; service degraded

Node : solaris10_node1.setaoffice.com
Node Type : Sun SPARC (HTTPS)
Severity : major
OM Server Time: 2015-06-23 09:22:58
Message : UXMON: genunix: [ID 408789 kern.warning] WARNING: ce1: fault detected external to device;

service degraded
Msg Group : OS
Application : SOL_mon
Object : hardware
Event Type :
not_found

Instance Name :
not_found

Instruction : “The Fault Management agent has identified a HW

or OS related problem with the severity presented by the ticket.
The problem(s) can be viewed and managed with the command – fmdump
To get a better understanding of the problem and on how to resolve it, locate the event that generated
the ticket in the syslog file /var/adm/messages, a URL will be found (http://sun.com/msg/xxx-nnnn-yy),
follow the link using your Oracle portal account for instructions.”

Check /var/adm/messages for mentions about this network interface

root@solaris10_node1:/ # grep ce1 /var/adm/messages
Jun 23 10:20:59 solaris10_node1 genunix: [ID 408789 kern.warning] WARNING: ce1: fault detected external to device; service degraded
Jun 23 10:20:59 solaris10_node1 genunix: [ID 451854 kern.warning] WARNING: ce1: xcvr addr:0x01 – link down

Also check dmesg

root@solaris10_node1:/ # dmesg | grep ce1
Jun 22 14:26:02 solaris10_node1 genunix: [ID 408789 kern.warning] WARNING: ce1: fault detected external to device; service degraded
Jun 22 14:26:02 solaris10_node1 genunix: [ID 451854 kern.warning] WARNING: ce1: xcvr addr:0x01 – link down
Jun 22 14:26:05 solaris10_node1 genunix: [ID 408789 kern.notice] NOTICE: ce1: fault cleared external to device; service available
Jun 22 14:26:05 solaris10_node1 genunix: [ID 451854 kern.notice] NOTICE: ce1: xcvr addr:0x01 – link up 10 Mbps half duplex
Jun 22 14:26:11 solaris10_node1 genunix: [ID 408789 kern.warning] WARNING: ce1: fault detected external to device; service degraded
Jun 22 14:26:11 solaris10_node1 genunix: [ID 451854 kern.warning] WARNING: ce1: xcvr addr:0x01 – link down
Jun 22 14:26:12 solaris10_node1 genunix: [ID 408789 kern.notice] NOTICE: ce1: fault cleared external to device; service available
Jun 22 14:26:12 solaris10_node1 genunix: [ID 451854 kern.notice] NOTICE: ce1: xcvr addr:0x01 – link up 10 Mbps half duplex
Jun 23 10:20:59 solaris10_node1 genunix: [ID 408789 kern.warning] WARNING: ce1: fault detected external to device; service degraded
Jun 23 10:20:59 solaris10_node1 genunix: [ID 451854 kern.warning] WARNING: ce1: xcvr addr:0x01 – link down

If you don’t need monitoring, configure /var/opt/OV/conf/OpC/dmsg_mon.cfg to suppress this alarm.

ia64dsk: The disk for dev_t d00009b appears to have grown since the partition table was written.

We are seeing a message about a disk to have grown since the partition table was written.

We had a storage migration using an HP MPX200 router

root@hp-ux:~ # dmesg

Mar 19 14:33
…
09b appears to have grown since the partition table was written.
ia64dsk: The disk for dev_t d00009b appears to have grown since the partition table was written.

To identify the disk, check /dev

root@hp-ux:~ # ls -ltraR /dev | grep 009b
crw-r—– 1 bin sys 13 0x00009b Feb 12 14:29 disk30
brw-r—– 1 bin sys 1 0x00009b Feb 12 14:29 disk30

According to idisk, we need to use -G option but it is not documented on the manual page

root@hp-ux:~ # idisk -p /dev/rdisk/disk30
idisk version: 1.44

EFI Primary Header:
Signature = EFI PART
Revision = 0x10000
HeaderSize = 0x5c
HeaderCRC32 = 0x65182f71
MyLbaLo = 0x1
MyLbaHi = 0x0
AlternateLbaLo = 0x117fffff
AlternateLbaHi = 0x0
FirstUsableLbaLo = 0x40
FirstUsableLbaHi = 0x0
LastUsableLbaLo = 0x117fffbf
LastUsableLbaHi = 0x0
Disk GUID = 9d7ea890-0835-11e4-8000-d6217b60e588
PartitionEntryLbaLo = 0x2
PartitionEntryLbaHi = 0x0
NumberOfPartitionEntries = 0xc
SizeOfPartitionEntry = 0x80
PartitionEntryArrayCRC32 = 0x6c45b3e0

Primary Partition Table (in 512 byte blocks):
Partition 1 (EFI):
Partition Type GUID = c12a7328-f81f-11d2-ba4b-00a0c93ec93b
Unique Partition GUID = 9d7eaaf2-0835-11e4-8000-d6217b60e588
Starting Lba Lo = 0x40
Starting Lba Hi = 0x0
Ending Lba Lo = 0xf9fff
Ending Lba Hi = 0x0
Partition 2 (HP-UX):
Partition Type GUID = 75894c1e-3aeb-11d3-b7c1-7b03a0000000
Unique Partition GUID = 9d7eab10-0835-11e4-8000-d6217b60e588
Starting Lba Lo = 0xfa000
Starting Lba Hi = 0x0
Ending Lba Lo = 0x117377ff
Ending Lba Hi = 0x0
Partition 3 (HPSP):
Partition Type GUID = e2a1e728-32e3-11d6-a682-7b03a0000000
Unique Partition GUID = 9d7eab24-0835-11e4-8000-d6217b60e588
Starting Lba Lo = 0x11737800
Starting Lba Hi = 0x0
Ending Lba Lo = 0x117ff7ff
Ending Lba Hi = 0x0
NOTE: This disk appears to have grown since the partition
table was written. Use idisk with the -G option to
extend the usable space to fill the disk.

There should be a difference in cylinders and blocks between the two disks that is causing this message