Tag: ovc

UXMON: The selfclean UXMON module has failed, please check reason on server

Node : solaris.setaoffice.com
Node Type : Sun SPARC (HTTPS)
Severity : warning
OM Server Time: 2019-05-02 02:00:04
Message : UXMON: The selfclean UXMON module has failed, please check reason
Msg Group : OS
Application : uxmon
Object : selfclean
Event Type :
not_found

Instance Name :
not_found

Instruction : No
EventDataSource :

Failure: At least one ITO agent process is not running on linux.setaoffice.com

I was having some problems with opcacta

root@linux:~ # ovc -status
coda OV Performance Core COREXT (23107) Running
opcacta OVO Action Agent AGENT,EA Aborted
opcle OVO Logfile Encapsulator AGENT,EA (23073) Running
opcmona OVO Monitor Agent AGENT,EA (6052) Running
opcmsga OVO Message Agent AGENT,EA (22896) Running
opcmsgi OVO Message Interceptor AGENT,EA (22979) Running
ovbbccb OV Communication Broker CORE (22700) Running
ovcd OV Control CORE (22687) Running
ovconfd OV Config and Deploy COREXT (22725) Running

Looking at /var/opt/OV/log/System.txt after stopping and starting, it shows a message saying that there is a problem with opcacta

root@linux:~ # tail -20 /var/opt/OV/log/System.txt
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:31 2016: opcacta (10806/47113196044032): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:31 2016: ovcd (10437/47099555100992): (ctrl-208) Component ‘opcacta’ with pid 10806 exited with exit value ‘0’. Restarting component.
0: ERR: Fri Apr 29 10:47:36 2016: ovcd (10437/47099477543232): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:36 2016: ovc (10434/47239787368608): (ctrl-7) Error in the target component.
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:36 2016: ovcd (10437/47099573193024): (ctrl-208) Component ‘opcacta’ with pid 12760 exited with exit value ‘0’. Restarting component.
0: INF: Fri Apr 29 10:47:37 2016: coda (10779/47049355707824): SCOPE datasource initialization succeeded
0: ERR: Fri Apr 29 10:47:41 2016: ovcd (10437/47099450292544): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: ERR: Fri Apr 29 10:47:42 2016: ovcd (10437/47099573193024): (ctrl-94) Component ‘opcacta’ exited after very short runtime, therefore it will not be automatically restarted. Use ‘ovc -start opcacta’.

Stop OVC

root@linux:~ # ovc -kill

Stop OVPA

root@linux:~ # ovpa stop

Shutting down Perf Agent collection software
Shutting down scopeux, pid(s) 13146
Waiting on 13146 (10 more tries)
The Perf Agent collector, scopeux has been shut down successfully.
NOTE: The ARM registration daemon ttd will be left running.

Shutting down the alarm generator perfalarm, pid(s) 13244
The perfalarm process has terminated

Remove log* files under /var/opt/perf/datafiles/

root@linux:~ # rm /var/opt/perf/datafiles/log*
rm: remove regular file `/var/opt/perf/datafiles/logappl’? y
rm: remove regular file `/var/opt/perf/datafiles/logdev’? y
rm: remove regular file `/var/opt/perf/datafiles/logglob’? y
rm: remove regular file `/var/opt/perf/datafiles/logindx’? y
rm: remove regular file `/var/opt/perf/datafiles/logpcmd0′? y
rm: remove regular file `/var/opt/perf/datafiles/logproc’? y
rm: remove regular file `/var/opt/perf/datafiles/logtran’? y

Remove *q files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*q
rm: remove regular file `/var/opt/OV/tmp/OpC/aaoorbCvq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/actagtq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpicmaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpimaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtq’? y

Remove *df files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*df
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtdf’? y

Restart HPOM and OVPA

root@linux:~ # /opt/perf/bin/ovpa start

The Perf Agent scope collector is being started.
The ARM registration daemon ttd is already running.
It will be signaled to reprocess its configuration file.

The Performance collection daemon
/opt/perf/bin/scopeux has been started.

The coda daemon /opt/OV/lbin/perf/coda has been started.
It will be fully operational in a few minutes.

The Perf Agent alarm generator is being started.
The alarm generator /opt/perf/bin/perfalarm
has been started.

root@linux:~ # ovc -start

root@linux:~ # ovc -status
coda OV Performance Core COREXT (22782) Running
opcacta OVO Action Agent AGENT,EA (23343) Running
opcle OVO Logfile Encapsulator AGENT,EA (23426) Running
opcmona OVO Monitor Agent AGENT,EA (23501) Running
opcmsga OVO Message Agent AGENT,EA (23238) Running
opcmsgi OVO Message Interceptor AGENT,EA (23459) Running
ovbbccb OV Communication Broker CORE (22755) Running
ovcd OV Control CORE (22747) Running
ovconfd OV Config and Deploy COREXT (22889) Running

HPOM certificate request: terminate called after throwing an instance of ‘char const*’

Triggered the certificate request but gave the following error

root@linux:~ # ovcert –certreq
terminate called after throwing an instance of ‘char const*’
Aborted

Stop HPOM and remove the files from the directories shown below. Verify if there is a HPOM agent process running and then kill if there is.

root@linux:~ # /opt/OV/bin/ovc -kill
root@linux:~ # /opt/OV/bin/opcagt -kill
(ctrl-111) Ovcd is not yet started.

root@linux:~ # rm /var/opt/OV/tmp/OpC/*
rm: cannot remove `/var/opt/OV/tmp/OpC/bin’: Is a directory
rm: cannot remove `/var/opt/OV/tmp/OpC/conf’: Is a directory
root@linux:~ # rm /var/opt/OV/tmp/public/OpC/*
rm: cannot remove `/var/opt/OV/tmp/public/OpC/*’: No such file or directory
root@linux:~ # rm /var/opt/OV/tmp/*.pid

root@linux:~ # ps -ef |grep -i opc
root@linux:~ # ps -ef |grep -i ov

Wait 3 minutes then start the agent

root@linux:~ # sleep 180

root@linux:~ # /opt/OV/bin/opcagt -start
root@linux:~ # /opt/OV/bin/ovc -start -debug

root@linux:~ # /opt/OV/bin/opcagt -status
scopeux Perf Agent data collector (23366) Running
midaemon Measurement Interface daemon (23372) Running
ttd ARM registration daemon (23354) Running
perfalarm Alarm generator (23421) Running
coda OV Performance Core COREXT (23414) Running
opcacta OVO Action Agent AGENT,EA (23565) Running
opcmsga OVO Message Agent AGENT,EA (23542) Running
ovbbccb OV Communication Broker CORE (23395) Running
ovcd OV Control CORE (23387) Running
ovconfd OV Config and Deploy COREXT (23507) Running
Message Agent is not buffering.

root@linux:~ # /opt/OV/bin/ovc -status
coda OV Performance Core COREXT (23414) Running
opcacta OVO Action Agent AGENT,EA (23565) Running
opcmsga OVO Message Agent AGENT,EA (23542) Running
ovbbccb OV Communication Broker CORE (23395) Running
ovcd OV Control CORE (23387) Running
ovconfd OV Config and Deploy COREXT (23507) Running

root@linux:~ # /opt/perf/bin/ovpa start

The Perf Agent scope collector is being started.
The ARM registration daemon ttd is already running.
It will be signaled to reprocess its configuration file.

The Performance Collector daemon
/opt/perf/bin/scopeux, is already running.

The coda daemon /opt/OV/lbin/perf/coda is already running.
The alarm generator /opt/perf/bin/perfalarm is already running.
It is signaled to reprocess its alarm definitions.

root@linux:~ # /opt/perf/bin/ovpa status
Perf Agent status:
Running scopeux (Perf Agent data collector) pid 23366
Running midaemon (Measurement Interface daemon) pid 23372
Running ttd (ARM registration daemon) pid 23354

Perf Agent Server status:

Running ovcd (OV control component) pid 23387
Running ovbbccb (BBC5 communication broker) pid 23395
Running coda (perf component) pid(s) 23414
Running perfalarm (alarm generator) pid(s) 23421

root@linux:~ # ovc -status
coda OV Performance Core COREXT (23414) Running
opcacta OVO Action Agent AGENT,EA (23565) Running
opcmsga OVO Message Agent AGENT,EA (23542) Running
ovbbccb OV Communication Broker CORE (23395) Running
ovcd OV Control CORE (23387) Running
ovconfd OV Config and Deploy COREXT (23507) Running

root@linux:~ # ovcert -certreq
INFO: Certificate request has been successfully triggered.