Category: Linux

Postfix not starting in Suse Linux

Postfix wasn’t starting

root@suse11sp2:~ # /etc/init.d/postfix start
Starting mail service (Postfix) failed

I ran in debug mode and found out which is the line to start postfix manually. I ran and it gave me the following error

root@suse11sp2:~ # /usr/sbin/postfix start
postsuper: fatal: scan_dir_push: open directory defer: Permission denied
postfix/postfix-script: fatal: Postfix integrity check failed!

To fix this error, run the following command

root@suse11sp2:~ # postfix -c /etc/postfix set-permissions
chown: cannot access `/usr/share/doc/packages/postfix-doc/README_FILES’: No such file or directory

I’m running Suse Linux 11 SP2 on this server

root@suse11sp2:~ # cat /etc/*release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 2
LSB_VERSION=”core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64″

It was not installed the package postfix-doc so I installed

root@suse11sp2:~ # rpm -Uvh postfix-doc-2.5.13-0.17.4.x86_64.rpm
Preparing… ########################################### [100%]
1:postfix-doc ########################################### [100%]

With the package installed, it didn’t give any error messages

root@suse11sp2:~ # postfix -c /etc/postfix set-permissions

And started successfully

root@suse11sp2:~ # /etc/init.d/postfix start
Starting mail service (Postfix) done

HPOM – UXMON: There were no valid peers!

I was receiving this incident from HPOM

Node : linuxnode.setaoffice.com
Node Type : Intel/AMD x64(HTTPS)
Severity : minor
OM Server Time: 2014-01-21 12:20:43
Message : UXMON: There were no valid peers!
Msg Group : OS
Application : ntpmon
Object : ntpq
Event Type :
not_found

Instance Name :
not_found

Instruction : This message shows athe ntpq has problems to contact
with the NTPD daemon. Is posible the NTP Daemon is down

Please, contact with your UX expert

Took a look at the log file and it was showing that is a problem with NTP

root@linuxnode:/ # tail /var/opt/OV/log/OpC/ntp_mon.log
Tue Jan 21 15:34:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 16:08:16 2014 : working peer “172.16.4.198”, offset(255.81) greater than NTP_OFFSET_WARNING(230), please check!
Tue Jan 21 16:59:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 17:16:17 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 17:33:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 18:41:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 19:49:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 20:06:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 21:14:16 2014 : NTP Problems. There were no valid peers!
Tue Jan 21 21:31:16 2014 : NTP Problems. There were no valid peers!

Apparently it is not synchronizing with more than one peer

root@linuxnode:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
172.23.72.50 .INIT. 16 u – 16 0 0.000 0.000 0.000
172.23.72.51 .INIT. 16 u – 16 0 0.000 0.000 0.000
172.23.72.52 .INIT. 16 u – 16 0 0.000 0.000 0.000
172.16.4.198 172.23.72.51 4 u 16 16 3 52.630 5.543 4.069

Edit /etc/ntp.conf and insert new NTP servers

root@linuxnode:~ #vi /etc/ntp.conf
#
#server 172.23.72.50 version 3 minpoll 4 maxpoll 4
server 142.40.238.18 version 3 minpoll 4 maxpoll 4
#server 172.23.72.51 version 3 minpoll 4 maxpoll 4
server 142.40.238.19 version 3 minpoll 4 maxpoll 4
#server 172.23.72.52 version 3 minpoll 4 maxpoll 4
server 142.40.236.20 version 3 minpoll 4 maxpoll 4
server 172.16.4.198 version 3 minpoll 4 maxpoll 4

Restart NTP

root@linuxnode:~ # /etc/init.d/ntp restart
Shutting down network time protocol daemon (NTPD) done
Try to get initial date and time via NTP from ntp.setaoffice.com done
Starting network time protocol daemon (NTPD) done

Then checking again the results

root@linuxnode:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
142.40.238.18 142.40.234.152 3 u 2 16 1 0.476 51.879 0.001
142.40.238.19 142.40.238.18 4 u 1 16 1 0.406 50.896 0.001
142.40.236.20 .INIT. 16 u – 16 0 0.000 0.000 0.001
172.16.4.198 .INIT. 16 u – 16 0 0.000 0.000 0.001

root@linuxnode:~ # ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
142.40.238.18 142.40.234.152 3 u 14 16 1 0.476 51.879 0.001
142.40.238.19 142.40.238.18 4 u 13 16 1 0.406 50.896 0.001
142.40.236.20 172.22.248.13 5 u 12 16 1 0.173 49.724 0.001
172.16.4.198 172.23.72.51 4 u 11 16 1 52.181 53.676 0.001

Trying to run ipmitool and receive an error where it can’t find the ipmi device

Running ipmitool was giving the following error

root@linux:~ # ipmitool lan print
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
Get Channel Info command failed
Invalid Channel 255

After loading the following kernel modules, the problem was solved

root@linux:~ # modprobe ipmi_msghandler
root@linux:~ # modprobe ipmi_devintf
root@linux:~ # modprobe ipmi_si

Running again ipmitool

root@linux:~ # ipmitool lan print
Set in Progress : Set Complete
Auth Type Support :
Auth Type Enable : Callback :
: User :
: Operator :
: Admin : MD2 PASSWORD OEM
: OEM : NONE MD2 PASSWORD
IP Address Source : Static Address
IP Address : 172.23.204.25
Subnet Mask : 255.255.255.0
MAC Address : 00:1e:0b:cb:77:5c
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 0.0 seconds
Default Gateway IP : 172.23.204.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
Cipher Suite Priv Max : Not Available

I was receiving this error message when trying to find the iLO IP address for a server:

root@linux:~ # modprobe ipmi_si
FATAL: Error inserting ipmi_si (/lib/modules/2.6.5-7.308-bigsmp/kernel/drivers/char/ipmi/ipmi_si.ko): No such device

And in /var/log/messages it shows that is unable to find any system interface.

root@linux:~ # tail -f /var/log/messages
May 8 11:13:26 linux kernel: IPMI System Interface driver.
May 8 11:13:26 linux kernel: IPMI System Interface driver.
May 8 11:13:26 linux kernel: ipmi_si: Trying “kcs” at I/O port 0xca2
May 8 11:13:26 linux kernel: ipmi_si: Trying “kcs” at I/O port 0xca2
May 8 11:13:26 linux kernel: ipmi_si: Trying “smic” at I/O port 0xca9
May 8 11:13:26 linux kernel: ipmi_si: Trying “smic” at I/O port 0xca9
May 8 11:13:26 linux kernel: ipmi_si: Trying “bt” at I/O port 0xe4
May 8 11:13:26 linux kernel: ipmi_si: Trying “bt” at I/O port 0xe4
May 8 11:13:26 linux kernel: ipmi_si: Unable to find any System Interface(s)
May 8 11:13:26 linux kernel: ipmi_si: Unable to find any System Interface(s)

root@linux:~ # cat /etc/*release
SUSE LINUX Enterprise Server 9 (i586)
VERSION = 9
PATCHLEVEL = 4
LSB_VERSION=”core-2.0-noarch:core-3.0-noarch:core-2.0-ia32:core-3.0-ia32″

This is a HP PROLIANT DL580 G3 and has a iLO v.1 so Linux says that is unable to find any System Interface. Since I’m unable to load IPMI, I was told to use an alternative method described here – Discover HP iLO IP address in Linux

Reset the password for the user Administrator in HP iLO 2

Create an XML file with the following contents

root@linux:~ # vi passwd_reset_ilo.xml
<ribcl VERSION=”2.0″>
<login USER_LOGIN=”Administrator” PASSWORD=”boguspassword”>
<user_INFO MODE=”write”>
<mod_USER USER_LOGIN=”Administrator”>
<password value=”newpassword”/>
</mod_USER>
</user_INFO>
</login>
</ribcl>

This file will reset the user Administrator with the password newpassword

Download hponcfg going here – http://www.hp.com/support/ilo2

Select the appropriate Linux operating system.
Select Software – Lights-Out Management category.
Select the HP Lights-Out Online Configuration Utility for Linux .
Select the appropriate version of the HP Lights-Out Online Configuration Utility and click the “download” button.

After installing, run as shown below.

root@linux:~ # hponcfg -f passwd_reset_ilo.xml -l log.txt
HP Lights-Out Online Configuration utility
Version 4.0.1 Date 09/24/2012 (c) Hewlett-Packard Company, 2012
Firmware Revision = 2.01 Device type = iLO 2 Driver name = hpilo
Script succeeded

Source – http://amal.net/?p=1675

Discover HP iLO IP address in Linux

Here we have a Proliant DL785 G6

root@linux:~ # dmidecode | grep -i Proliant
Product Name: ProLiant DL785 G6
Family: ProLiant

To discover the iLO IP address, check it with ipmitool

root@linux:~ # ipmitool lan print
Set in Progress : Set Complete
Auth Type Support :
Auth Type Enable : Callback :
: User :
: Operator :
: Admin : MD2 PASSWORD OEM
: OEM : NONE MD2 PASSWORD
IP Address Source : Static Address
IP Address : 172.22.29.40
Subnet Mask : 255.255.255.0
MAC Address : 00:21:5a:de:db:02
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 0.0 seconds
Default Gateway IP : 172.22.29.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
Cipher Suite Priv Max : Not Available

Or you can view using hponcfg and outputting the configuration to a file

root@linux:~ # hponcfg -w iLO_ouput.out
root@linux:~ # cat ILO_ouput.out
<!– HPONCFG VERSION = “3.0” –>
<!– Generated 5/8/2014 12:6:50 –>
<DIR_INFO MODE=”write”>
<MOD_DIR_CONFIG>
<DIR_AUTHENTICATION_ENABLED VALUE = “N”/>
<DIR_LOCAL_USER_ACCT VALUE = “Y”/>
<DIR_SERVER_ADDRESS VALUE = “”/>
<DIR_SERVER_PORT VALUE = “636”/>
<DIR_OBJECT_DN VALUE = “”/>
<DIR_OBJECT_PASSWORD VALUE = “”/>
<DIR_USER_CONTEXT_1 VALUE = “”/>
<DIR_USER_CONTEXT_2 VALUE = “”/>
<DIR_USER_CONTEXT_3 VALUE = “”/>
<RIB_INFO MODE=”write”>
<MOD_NETWORK_SETTINGS>
<SPEED_AUTOSELECT VALUE = “Y”/>
<NIC_SPEED VALUE = “10”/>
<FULL_DUPLEX VALUE = “N”/>
<IP_ADDRESS VALUE = “0.0.0.0”/>
<SUBNET_MASK VALUE = “0.0.0.0”/>
<GATEWAY_IP_ADDRESS VALUE = “0.0.0.0”/>
<DNS_NAME VALUE = “ILO—z”/>
<PRIM_DNS_SERVER value = “0.0.0.0”/>
<DHCP_ENABLE VALUE = “Y”/>
<DOMAIN_NAME VALUE = “”/>
<DHCP_GATEWAY VALUE = “Y”/>
<DHCP_DNS_SERVER VALUE = “Y”/>
<DHCP_STATIC_ROUTE VALUE = “Y”/>
<DHCP_WINS_SERVER VALUE = “Y”/>
<REG_WINS_SERVER VALUE = “Y”/>
<PRIM_WINS_SERVER value = “0.0.0.0”/>
<STATIC_ROUTE_1 DEST = “0.0.0.0” GATEWAY = “0.0.0.0”/>
<STATIC_ROUTE_2 DEST = “0.0.0.0” GATEWAY = “0.0.0.0”/>
<STATIC_ROUTE_3 DEST = “0.0.0.0” GATEWAY = “0.0.0.0”/>
</MOD_NETWORK_SETTINGS>
</RIB_INFO>
<USER_INFO MODE=”write”>
</USER_INFO>
</LOGIN>
</RIBCL>

Related: Trying to run ipmitool and receive an error where it can’t find the ipmi device

IBM Rational ClearCase mount.mvfs: Too many open files in system

There is a Linux host where ClearCase can’t mount any more VOBs

mount.mvfs: Too many open files in system

Reading this note on http://www-01.ibm.com/support/docview.wss?uid=swg21234500#linux says the following:
The number of VOBs that can be mounted simultaneously under Linux is 1022, but this limit can be increased by configuring your MVFS. After you edit the mvfs_param.c file as described below for the kernel version of your system, rebuild and reinstall the MVFS or vnode module using the instructions contained in the README.txt file in the directory containing mvfs_params.c.

This is exactly the case on my system

root@linux:~ # df -h | grep vbs | wc -l
1022

Linux server with “vpath” disks

I recently saw a Linux server connected to an IBM storage with a “vpath” disk. It is because the server uses IBM Subsystem Device Driver.

Reading the IBM System Storage Multipath Subsystem Device Driver (SDD) User’s Guide, I figured how to check the disks

root@linux:~ # datapath query device

Total Devices : 47

DEV#: 0 DEVICE NAME: vpathb TYPE: 2145 POLICY: Optimized Sequential
SERIAL: 600507680191818c1000000000000304
============================================================================
Path# Adapter/Hard Disk State Mode Select Errors
0 Host0Channel0/sda OPEN NORMAL 2059 0
1 Host0Channel0/sday OPEN NORMAL 3761 0
2 Host0Channel0/sdcw OPEN NORMAL 0 0
3 Host0Channel0/sdeu OPEN NORMAL 0 0
4 Host1Channel0/sdgs OPEN NORMAL 3122 0
5 Host1Channel0/sdiq OPEN NORMAL 2949 0
6 Host1Channel0/sdko OPEN NORMAL 0 0
7 Host1Channel0/sdmm OPEN NORMAL 0 0

Error: pvcreate: Device /dev/mapper/software01p1 not found

When trying to run pvcreate on a partition that exists on the system, it says that device is not found.

root@linux:/ # pvcreate /dev/mapper/software01p1
Device /dev/mapper/software01p1 not found.

The device exists on the system

root@linux:/ # ls -l /dev/mapper/software01*
brw——- 1 root root 253, 7 Jul 27 10:23 /dev/mapper/software01
brw——- 1 root root 253, 8 Jul 27 10:27 /dev/mapper/software01p1

The problem was that the LVM filter is not configured to consider it as a valid device.

Edit the file /etc/lvm/lvm.conf and edit the parameter filter accordingly. I used the filter below:

root@linux:/ # vi /etc/lvm/lvm.conf
filter = [ “a|cciss/.*|” “a|/dev/mapper/.*|”, “r|/dev/sd.*|”, “r|/dev/dm-.*|” ]

Querying HBA information under Linux

To check the HBA cards installed on your system, check lspci

root@linux:~ # lspci | grep -i fibre
13:01.0 Fibre Channel: Emulex Corporation Thor-X LightPulse Fibre Channel Host Adapter (rev 01)
13:02.0 Fibre Channel: Emulex Corporation Thor-X LightPulse Fibre Channel Host Adapter (rev 01)

You can query various information using systool

root@linux:~ # systool -av -c fc_host
Class = “fc_host”

Class Device = “host0”
Class Device path = “/sys/class/fc_host/host0”
active_fc4s = “0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ”
fabric_name = “0x100000051eecc200”
issue_lip =
maxframe_size = “2048 bytes”
node_name = “0x20000000c95e5ee1”
port_id = “0x0ab040”
port_name = “0x10000000c95e5ee1”
port_state = “Online”
port_type = “NPort (fabric via point-to-point)”
speed = “2 Gbit”
supported_classes = “Class 3”
supported_fc4s = “0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ”
supported_speeds = “1 Gbit, 2 Gbit”
tgtid_bind_type = “wwpn (World Wide Port Name)”
uevent =

Device = “host0”
Device path = “/sys/devices/pci0000:00/0000:00:04.0/0000:12:00.0/0000:13:01.0/host0”
uevent =

Class Device = “host1”
Class Device path = “/sys/class/fc_host/host1”
active_fc4s = “0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ”
fabric_name = “0x100000051ef63200”
issue_lip =
maxframe_size = “2048 bytes”
node_name = “0x20000000c95e93bd”
port_id = “0x14b040”
port_name = “0x10000000c95e93bd”
port_state = “Online”
port_type = “NPort (fabric via point-to-point)”
speed = “2 Gbit”
supported_classes = “Class 3”
supported_fc4s = “0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ”
supported_speeds = “1 Gbit, 2 Gbit”
tgtid_bind_type = “wwpn (World Wide Port Name)”
uevent =

Device = “host1”
Device path = “/sys/devices/pci0000:00/0000:00:04.0/0000:12:00.0/0000:13:02.0/host1”
uevent =

Checking numerous status. Example shown below is a different server

root@linux:~ # systool -c scsi_host -v -d host1
Class = “scsi_host”

Class Device = “host1”
Class Device path = “/sys/class/scsi_host/host1”
84xx_fw_version = “”
beacon = “Disabled”
cmd_per_lun = “3”
driver_version = “8.03.07.15.05.09-k”
eh_deadline = “0”
fabric_param = “1280”
flash_block_size = “0x8000”
fw_state = “0x3 0x2a8c 0x9100 0x5 0x0”
fw_version = “5.06.03 (9496)”
host_busy = “0”
isp_id = “0000 0000 0000 0000”
isp_name = “ISP2432”
max_npiv_vports = “127”
model_desc = “PCI-Express Dual Channel 4Gb Fibre Channel HBA”
model_name = “QLE2462”
mpi_version = “”
npiv_vports_inuse = “0”
optrom_bios_version = “1.08”
optrom_efi_version = “1.02”
optrom_fcode_version= “1.13”
optrom_fw_version = “4.00.12 2”
pci_info = “PCIe (2.5Gb/s x4)”
phy_version = “”
proc_name = “”
scan =
serial_num = “”
sg_tablesize = “255”
state = “Link Up – F_Port”
total_isp_aborts = “0”
uevent =
unchecked_isa_dma = “0”
unique_id = “0”
vlan_id = “”
vn_port_mac_address = “”
vport_create =
vport_delete =
zio_timer = “200 us”
zio = “Disabled”

Device = “host1”
Device path = “/sys/devices/pci0000:00/0000:00:04.0/0000:17:00.0/0000:18:02.0/0000:22:00.0/host1”
fw_dump =
nvram = “ISP ”
optrom_ctl =
optrom =
reset =
sfp = “”
uevent =
vpd = “▒.”

Or you can query each attribute issuing cat /sys/class/scsi_host/hostX/<attribute>

Model name

root@linux:~ # cat /sys/class/scsi_host/host0/model*name
LP10000

Model description

root@linux:~ # cat /sys/class/scsi_host/host0/model*desc
Emulex LP10000 2Gb PCI-X Fibre Channel Adapter

Check this link to see how to upgrade a Fibre Channel HBA on HP Proliant – Upgrading firmware on an Emulex Zephyr-X LightPulse Fibre Channel HBA

HBA ROM version

root@linux:~ # cat /sys/class/scsi_host/host0/opt*rom*
5.01a5

Firmware revision

root@linux:~ # cat /sys/class/scsi_host/host0/fwrev
1.91A1 (T2D1.91A1), sli-2

root@linux:~ # cat /sys/class/scsi_host/host0/fw_version
5.03.01 (496)

Driver version

root@linux:~ # lspci -v | grep -i fibre -A11 | grep driver
Kernel driver in use: lpfc
Kernel driver in use: lpfc

root@linux:~ # modinfo lpfc | egrep ‘filename|version|author|description’
filename: /lib/modules/2.6.18-238.9.1.el5/kernel/drivers/scsi/lpfc/lpfc.ko
version: 0:8.2.0.87.1p
author: Emulex Corporation – tech.support@emulex.com
description: Emulex LightPulse Fibre Channel SCSI driver 8.2.0.87.1p

View script to query HBA information

Error updating RHEL package: Cannot retrieve repository metadata for repository: rhel-x86_64-server-5

I made a local repository on my network to avoid updating the package through the internet. But my system was giving me the following error

root@linux:~ # yum update nscd
Loaded plugins: rhnplugin, security
Error: Cannot retrieve repository metadata (repomd.xml) for repository: rhel-x86_64-server-5. Please verify its path and try again

To solve this problem, I disabled the rhnplugin

root@linux:~ # vi /etc/yum/pluginconf.d/rhnplugin.conf
[main]
enabled = 0
gpgcheck = 1

# You can specify options per channel, e.g.:
#
#[rhel-i386-server-5]
#enabled = 1
#
#[some-unsigned-custom-channel]
#gpgcheck = 0