Failure: At least one ITO agent process is not running on linux.setaoffice.com

I was having some problems with opcacta

root@linux:~ # ovc -status
coda OV Performance Core COREXT (23107) Running
opcacta OVO Action Agent AGENT,EA Aborted
opcle OVO Logfile Encapsulator AGENT,EA (23073) Running
opcmona OVO Monitor Agent AGENT,EA (6052) Running
opcmsga OVO Message Agent AGENT,EA (22896) Running
opcmsgi OVO Message Interceptor AGENT,EA (22979) Running
ovbbccb OV Communication Broker CORE (22700) Running
ovcd OV Control CORE (22687) Running
ovconfd OV Config and Deploy COREXT (22725) Running

Looking at /var/opt/OV/log/System.txt after stopping and starting, it shows a message saying that there is a problem with opcacta

root@linux:~ # tail -20 /var/opt/OV/log/System.txt
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:31 2016: opcacta (10806/47113196044032): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:31 2016: ovcd (10437/47099555100992): (ctrl-208) Component ‘opcacta’ with pid 10806 exited with exit value ‘0’. Restarting component.
0: ERR: Fri Apr 29 10:47:36 2016: ovcd (10437/47099477543232): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:36 2016: ovc (10434/47239787368608): (ctrl-7) Error in the target component.
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:36 2016: opcacta (12760/47153555500800): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: WRN: Fri Apr 29 10:47:36 2016: ovcd (10437/47099573193024): (ctrl-208) Component ‘opcacta’ with pid 12760 exited with exit value ‘0’. Restarting component.
0: INF: Fri Apr 29 10:47:37 2016: coda (10779/47049355707824): SCOPE datasource initialization succeeded
0: ERR: Fri Apr 29 10:47:41 2016: ovcd (10437/47099450292544): (ctrl-42) Initialization of component ‘opcacta’ failed. Stopping component.
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxproc.c:476]: semget(2) failed; cannot create semaphore
(OpC20-415)
OpC internal error: Cannot generate message 435 of set 20
(OpC20-435)
0: ERR: Fri Apr 29 10:47:41 2016: opcacta (13439/47066407739136): [uxacta.c:524]: counter for critical events exceeded limit (counter value = 1) (OpC30-526)
0: ERR: Fri Apr 29 10:47:42 2016: ovcd (10437/47099573193024): (ctrl-94) Component ‘opcacta’ exited after very short runtime, therefore it will not be automatically restarted. Use ‘ovc -start opcacta’.

Stop OVC

root@linux:~ # ovc -kill

Stop OVPA

root@linux:~ # ovpa stop

Shutting down Perf Agent collection software
Shutting down scopeux, pid(s) 13146
Waiting on 13146 (10 more tries)
The Perf Agent collector, scopeux has been shut down successfully.
NOTE: The ARM registration daemon ttd will be left running.

Shutting down the alarm generator perfalarm, pid(s) 13244
The perfalarm process has terminated

Remove log* files under /var/opt/perf/datafiles/

root@linux:~ # rm /var/opt/perf/datafiles/log*
rm: remove regular file `/var/opt/perf/datafiles/logappl’? y
rm: remove regular file `/var/opt/perf/datafiles/logdev’? y
rm: remove regular file `/var/opt/perf/datafiles/logglob’? y
rm: remove regular file `/var/opt/perf/datafiles/logindx’? y
rm: remove regular file `/var/opt/perf/datafiles/logpcmd0′? y
rm: remove regular file `/var/opt/perf/datafiles/logproc’? y
rm: remove regular file `/var/opt/perf/datafiles/logtran’? y

Remove *q files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*q
rm: remove regular file `/var/opt/OV/tmp/OpC/aaoorbCvq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/actagtq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpicmaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/mpimaq’? y
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtq’? y

Remove *df files under /var/opt/OV/tmp/OpC/

root@linux:~ # rm /var/opt/OV/tmp/OpC/*df
rm: remove regular file `/var/opt/OV/tmp/OpC/msgagtdf’? y

Restart HPOM and OVPA

root@linux:~ # /opt/perf/bin/ovpa start

The Perf Agent scope collector is being started.
The ARM registration daemon ttd is already running.
It will be signaled to reprocess its configuration file.

The Performance collection daemon
/opt/perf/bin/scopeux has been started.

The coda daemon /opt/OV/lbin/perf/coda has been started.
It will be fully operational in a few minutes.

The Perf Agent alarm generator is being started.
The alarm generator /opt/perf/bin/perfalarm
has been started.

root@linux:~ # ovc -start

root@linux:~ # ovc -status
coda OV Performance Core COREXT (22782) Running
opcacta OVO Action Agent AGENT,EA (23343) Running
opcle OVO Logfile Encapsulator AGENT,EA (23426) Running
opcmona OVO Monitor Agent AGENT,EA (23501) Running
opcmsga OVO Message Agent AGENT,EA (23238) Running
opcmsgi OVO Message Interceptor AGENT,EA (23459) Running
ovbbccb OV Communication Broker CORE (22755) Running
ovcd OV Control CORE (22747) Running
ovconfd OV Config and Deploy COREXT (22889) Running

Advertisement