Emulex OneConnect initialization failed

I have a HP Proliant BL685C G7 and both network interfaces are missing.

Rebooted the server and it showed some messages from the network card

be2net 0000:05:00.0 Waiting for POST, 58s elapsed
be2net 0000:05:00.0 POST timeout; stage=0xb00
be2net 0000:05:00.0 PCI INT A disabled
be2net 0000:05:00.0 Emulex OneConnect initialization failed
be2net: probe of 0000:05:00.0 failed with error -1
ACPI: PCI Interrupt Link [I031] enabled at IRQ 29

According to this document HP Store All Storage – Interface ETH 4 and ETH 5 are Missing it is a physical hardware failure and you need to contact HP – even though that the affected equipment is a storage I believe it is the same case with the server.

There is also another document be2net Driver Fails to Initialize where it shows that the NIC fails to initiate and asks you to update to update to a newer kernel or upgrade the NIC firmware. In this case the server’s OS was upgraded recently and firmware updated.

be2net 0000:10:00.0: Emulex OneConnect 10Gbps NIC initialization failed
be2net: probe of 0000:10:00.0 failed with error -1

hpacucli takes a long time to respond

If run hpacucli and your server is taking a long time to respond, it is because HP decided to change the hpacucli utility to scan multiple LUNs connected via Fibre or iSCSI storage

root@linux:~ # time hpacucli ctrl all show config

Smart Array P410i in Slot 0 (Embedded) (sn: 500143801176C960)

array A (SAS, Unused Space: 0 MB)

logicaldrive 1 (136.7 GB, RAID 1, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 500143801176C96F)

real 2m25.058s
user 0m0.083s
sys 0m0.071s

Type INFOMGR_BYPASS_NONSA=1 to disable the scan hpacucli. To reenable type export -n INFOMGR_BYPASS_NONSA

root@linux:~# export INFOMGR_BYPASS_NONSA=1
root@linux:~# time hpacucli ctrl all show config

Smart Array P410i in Slot 0 (Embedded) (sn: 50014380226F2BA0)

array A (SAS, Unused Space: 0 MB)

logicaldrive 1 (279.4 GB, RAID 1, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 300 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 300 GB, OK)

SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 50014380226F2BAF)

real 0m0.931s
user 0m0.130s
sys 0m0.167s

Source: http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03696601

Removing vpath disks coming from IBM storage

root@linux:~ # datapath query device

Total Devices : 7

DEV#: 0 DEVICE NAME: vpathk TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000097
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sdaw CLOSE NORMAL 0 0
1 Host1Channel0/sdbd CLOSE NORMAL 0 0
2 Host1Channel0/sdai CLOSE NORMAL 173 0
3 Host1Channel0/sdap CLOSE NORMAL 106 0
4 Host0Channel0/sdu CLOSE NORMAL 0 0
5 Host0Channel0/sdab CLOSE NORMAL 0 0
6 Host0Channel0/sdg CLOSE NORMAL 80 0
7 Host0Channel0/sdn CLOSE NORMAL 236 0

DEV#: 1 DEVICE NAME: vpathn TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000092
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sdax CLOSE NORMAL 0 0
1 Host1Channel0/sdaj CLOSE NORMAL 9294 0
2 Host1Channel0/sdaq CLOSE NORMAL 0 0
3 Host0Channel0/sdv CLOSE NORMAL 0 0
4 Host1Channel0/sdac CLOSE NORMAL 9067 0
5 Host0Channel0/sda CLOSE NORMAL 8218 0
6 Host0Channel0/sdh CLOSE NORMAL 7460 0
7 Host0Channel0/sdo CLOSE NORMAL 0 0

DEV#: 2 DEVICE NAME: vpathl TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000096
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sday CLOSE NORMAL 1819009 0
1 Host1Channel0/sdak CLOSE NORMAL 0 0
2 Host1Channel0/sdar CLOSE NORMAL 1818549 0
3 Host0Channel0/sdw CLOSE NORMAL 1820897 0
4 Host1Channel0/sdad CLOSE NORMAL 0 0
5 Host0Channel0/sdb CLOSE NORMAL 0 0
6 Host0Channel0/sdi CLOSE NORMAL 0 0
7 Host0Channel0/sdp CLOSE NORMAL 1804045 0

DEV#: 3 DEVICE NAME: vpathm TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000091
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sdaz CLOSE NORMAL 0 0
1 Host1Channel0/sdal CLOSE NORMAL 100105 0
2 Host1Channel0/sdas CLOSE NORMAL 0 0
3 Host0Channel0/sdq CLOSE NORMAL 0 0
4 Host0Channel0/sdx CLOSE NORMAL 0 0
5 Host1Channel0/sdae CLOSE NORMAL 95597 0
6 Host0Channel0/sdc CLOSE NORMAL 96473 0
7 Host0Channel0/sdj CLOSE NORMAL 95534 0

DEV#: 4 DEVICE NAME: vpathh TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000095
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sdba CLOSE NORMAL 160 0
1 Host1Channel0/sdam CLOSE NORMAL 0 0
2 Host1Channel0/sdat CLOSE NORMAL 219 0
3 Host0Channel0/sdr CLOSE NORMAL 254 0
4 Host0Channel0/sdy CLOSE NORMAL 370 0
5 Host1Channel0/sdaf CLOSE NORMAL 0 0
6 Host0Channel0/sdd CLOSE NORMAL 0 0
7 Host0Channel0/sdk CLOSE NORMAL 0 0

DEV#: 5 DEVICE NAME: vpathi TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000094
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sdbb CLOSE NORMAL 4959899 0
1 Host1Channel0/sdag CLOSE NORMAL 0 0
2 Host1Channel0/sdan CLOSE NORMAL 0 0
3 Host1Channel0/sdau CLOSE NORMAL 4956162 0
4 Host0Channel0/sds CLOSE NORMAL 5032326 0
5 Host0Channel0/sdz CLOSE NORMAL 5055897 0
6 Host0Channel0/sde CLOSE NORMAL 0 0
7 Host0Channel0/sdl CLOSE NORMAL 0 0

DEV#: 6 DEVICE NAME: vpathj TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000093
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host1Channel0/sdbc CLOSE NORMAL 3715251 0
1 Host1Channel0/sdah CLOSE NORMAL 0 0
2 Host1Channel0/sdao CLOSE NORMAL 0 0
3 Host1Channel0/sdav CLOSE NORMAL 3743916 0
4 Host0Channel0/sdt CLOSE NORMAL 3823190 0
5 Host0Channel0/sdaa CLOSE NORMAL 3800038 0
6 Host0Channel0/sdf CLOSE NORMAL 0 0
7 Host0Channel0/sdm CLOSE NORMAL 0 0

Use the command rmvpath to remove all paths

root@linux:~ # rmvpath
Continuing will remove all IBM vpath devices.
Do you want to continue (y/n)?: n

This is a little man page for the rmvpath command

root@linux:~ # rmvpath ?

NAME
rmvpath — remove IBM vpath devices.

SYNOPSIS
rmvpath [OPTION]… [VPATH]

DESCRIPTION
rmvpath can be invoked with no options or with the name
of a specific vpath device to remove, and/or a specific
configuration file.

OPTIONS
-f [FILENAME] indicate a specific configuration file to update
rmvpath will default to /etc/vpath.conf
[VPATH] if no vpath is specified, all IBM vpath
devices will be removed
debug prints debug information to stdout

DEFAULTS
o default configuration file is /etc/vpath.conf
o all vpaths are removed

To remove /dev/vpath4, updating /tmp/configuration:

rmvpath -f /tmp/configuration vpath4

I removed only the vpathi disk here

root@linux:~ # rmvpath vpathi
Remove vpathi requested
Deleting file: /dev/vpathi

Source: Implementing Linux with IBM Disk Storage

Updating iLO3 firmware on a Proliant BL685c G7 running RHEL 5

I have a ProLiant BL685c G7 and I will update iLO3 firmware

Go to http://www8.hp.com/us/en/drivers.html
Search by the product name ProLiant BL685c G7
Select your operating system. In my case Red Hat Linux Enterprise 5 Server (x86-64)
Select Firmware – Lights-Out Management and download the file

Copy the file to the server and make it executable
root@linux:~ # chmod 700 CP022551.scexe

iLO3 was running version 1.55 and was upgraded to 1.80

root@linux:~ # ./CP022551.scexe

FLASH_iLO3 v1.09 for Linux (Jan 23 2013)
(C) Copyright 2002-2013 Hewlett-Packard Development Company, L.P.
Firmware image: ilo3_180.bin
Current iLO 3 firmware version 1.55; Serial number ILOUSE253RVDV

Component XML file: CP022551.xml
CP022551.xml reports firmware version 1.80
This operation will update the firmware on the
iLO 3 in this server with version 1.80.
Continue (y/N)?y
Current firmware is 1.55 (Jan 24 2013 )
Firmware image is 0x801664(8394340) bytes
Committing to flash part…
******** DO NOT INTERRUPT! ********
Flashing completed.
Attempting to reset device.
Succeeded.
***** iLO 3 reboot in progress (may take up to 60 seconds.)
***** Please ignore console messages, if any.
iLO 3 reboot completed.

Sun Cluster 3.1: Listener probe timed out two times consecutively

Oct 19 22:01:02 solaris10node1 SC[SUNWscor.oracle_listener.monitor]:dbs-oracle-proddb-rg:lsn-oraproddb-01-res: [ID 523531 user.error] Listener probe timed out two times consecutively

This error was appearing in one server because every night a full database export was being done. The listener was being overloaded.

Upgrading firmware on an Emulex Zephyr-X LightPulse Fibre Channel HBA on HP Proliant

I have a Linux server and I need to upgrade the HBA card.

First we verify which model is the HBA card. It’s a dual port HBA card (same addressing but the last number differ)

root@linux:~ # lspci -k | grep -A2 -i fibre
06:00.0 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
Subsystem: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter
Kernel driver in use: lpfc
Kernel modules: lpfc
06:00.1 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
Subsystem: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter
Kernel driver in use: lpfc
Kernel modules: lpfc

Verify the model description. It’s a HP FC2242SR 4Gb PCI-e DC Fibre Channel Adapter

root@linux:~ # cat /sys/class/scsi_host/host0/modeldesc
HP FC2242SR 4Gb PCI-e DC Fibre Channel Adapter
root@linux:~ # cat /sys/class/scsi_host/host1/modeldesc
HP FC2242SR 4Gb PCI-e DC Fibre Channel Adapter

Go to http://www8.hp.com/us/en/drivers.html

Type FC2242SR

Choose your operating system and download the scexe file under the box Firmware – Storage Fibre Channel

Copy the file to the server and then run the file you just downloaded

root@linux:~ # ./CP022701.scexe

HPOM certificate request: terminate called after throwing an instance of ‘char const*’

Triggered the certificate request but gave the following error

root@linux:~ # ovcert –certreq
terminate called after throwing an instance of ‘char const*’
Aborted

Stop HPOM and remove the files from the directories shown below. Verify if there is a HPOM agent process running and then kill if there is.

root@linux:~ # /opt/OV/bin/ovc -kill
root@linux:~ # /opt/OV/bin/opcagt -kill
(ctrl-111) Ovcd is not yet started.

root@linux:~ # rm /var/opt/OV/tmp/OpC/*
rm: cannot remove `/var/opt/OV/tmp/OpC/bin': Is a directory
rm: cannot remove `/var/opt/OV/tmp/OpC/conf': Is a directory
root@linux:~ # rm /var/opt/OV/tmp/public/OpC/*
rm: cannot remove `/var/opt/OV/tmp/public/OpC/*': No such file or directory
root@linux:~ # rm /var/opt/OV/tmp/*.pid

root@linux:~ # ps -ef |grep -i opc
root@linux:~ # ps -ef |grep -i ov

Wait 3 minutes then start the agent

root@linux:~ # sleep 180

root@linux:~ # /opt/OV/bin/opcagt -start
root@linux:~ # /opt/OV/bin/ovc -start -debug

root@linux:~ # /opt/OV/bin/opcagt -status
scopeux Perf Agent data collector (23366) Running
midaemon Measurement Interface daemon (23372) Running
ttd ARM registration daemon (23354) Running
perfalarm Alarm generator (23421) Running
coda OV Performance Core COREXT (23414) Running
opcacta OVO Action Agent AGENT,EA (23565) Running
opcmsga OVO Message Agent AGENT,EA (23542) Running
ovbbccb OV Communication Broker CORE (23395) Running
ovcd OV Control CORE (23387) Running
ovconfd OV Config and Deploy COREXT (23507) Running
Message Agent is not buffering.

root@linux:~ # /opt/OV/bin/ovc -status
coda OV Performance Core COREXT (23414) Running
opcacta OVO Action Agent AGENT,EA (23565) Running
opcmsga OVO Message Agent AGENT,EA (23542) Running
ovbbccb OV Communication Broker CORE (23395) Running
ovcd OV Control CORE (23387) Running
ovconfd OV Config and Deploy COREXT (23507) Running

root@linux:~ # /opt/perf/bin/ovpa start

The Perf Agent scope collector is being started.
The ARM registration daemon ttd is already running.
It will be signaled to reprocess its configuration file.

The Performance Collector daemon
/opt/perf/bin/scopeux, is already running.

The coda daemon /opt/OV/lbin/perf/coda is already running.
The alarm generator /opt/perf/bin/perfalarm is already running.
It is signaled to reprocess its alarm definitions.

root@linux:~ # /opt/perf/bin/ovpa status
Perf Agent status:
Running scopeux (Perf Agent data collector) pid 23366
Running midaemon (Measurement Interface daemon) pid 23372
Running ttd (ARM registration daemon) pid 23354

Perf Agent Server status:

Running ovcd (OV control component) pid 23387
Running ovbbccb (BBC5 communication broker) pid 23395
Running coda (perf component) pid(s) 23414
Running perfalarm (alarm generator) pid(s) 23421

root@linux:~ # ovc -status
coda OV Performance Core COREXT (23414) Running
opcacta OVO Action Agent AGENT,EA (23565) Running
opcmsga OVO Message Agent AGENT,EA (23542) Running
ovbbccb OV Communication Broker CORE (23395) Running
ovcd OV Control CORE (23387) Running
ovconfd OV Config and Deploy COREXT (23507) Running

root@linux:~ # ovcert -certreq
INFO: Certificate request has been successfully triggered.

Check HPOM version

root@linux:~ # opcagt -version
11.13.007

HP Data Protector error: [12:1625] Import host failed.

root@hp-ux:~ # omnicc -import_host suse11host.bkp
[12:1625] Import host failed.

The host information cannot be added/updated in the Cell Manager configuration file.

* Please check that the Data Protector services on the Cell Manager computer are running.
* Also check that Data Protector Inet service on the remote computer is running
(use: telnet 5555 to see if the service responds).
* Make sure that the remote client is not already part of another cell.
Afterwards use Data Protector GUI or omnicc -import_host command to manually import
the client to the selected Cell Manager.
* On NetWare make sure Data Protector directory structure is created properly
(check that directory SYS:USR\OMNI\CONFIG\CLIENT exists) or check that
localhost (127.0.0.1) entry exists in SYS:ETC\HOSTS file for servers
not using DNS IP address resolution.

[12:1051] Client security violation. Access denied.
, error=1625

Verify if file /etc/opt/omni/client/allow_hosts exists and if there is any content. Modify accordingly to your servers.

Also verify the log file from the inetd to verify the error

root@linux:/etc/xinetd.d # cat omni
service omni
{
socket_type = stream
protocol = tcp
wait = no
user = root
server = /opt/omni/lbin/inet
server_args = inet -log /var/opt/omni/log/inet.log
disable = no

instances = UNLIMITED
per_source = UNLIMITED
}

root@linux:~ # tail /var/opt/omni/log/inet.log
10/14/14 14:52:38 INET.20969.0 [“/inet/allow_deny.c $Rev: 29682 $ $Date:: 2012-03-15 19:42:59″:490] A.06.20 b409
A request 0 (INCLUDE) came from host hp-ux (172.22.19.188) which is not in AllowList: not proceeding with this request!
[Critical] From: INET@linux “” Time: 10/14/14 14:52:38
[12:1051] Client security violation. Access denied.

SSH connection error – no supported authentication methods available (server sent keyboard-interactive)

Edit /etc/ssh/sshd_config and set the parameter PAMAuthenticationViaKBDInt to yes

# To disable tunneled clear text passwords, change PasswordAuthentication to no.
PasswordAuthentication yes

# Use PAM via keyboard interactive method for authentication.
# Depending on the setup of pam.conf(4) this may allow tunneled clear text
# passwords even when PasswordAuthentication is set to no. This is dependent
# on what the individual modules request and is out of the control of sshd
# or the protocol.
#PAMAuthenticationViaKBDInt no
PAMAuthenticationViaKBDInt yes

Follow

Get every new post delivered to your Inbox.

Join 1,096 other followers

%d bloggers like this: