Nagios Performance Data shows checks aren't being completed

[Kate Harris]

I had a similar problem and thought I had fixed it.

My situation is that I have 922 services to check (at the moment, I need to
ramp up to over 2,500 but the latency problem is a show-stopper at the
moment).  I'm using a very low-spec Dell running Solaris 10 with
Nagios 2.0to do it.  Using default settings, I was initially getting
average check
latencies of the order of 5-6 seconds which was fine, but after a day or so
of no Nagios restarts, that figure would rocket to 100 seconds and stay
there, not ever re-checking the majority of the services, with re-scheduled
check times staying in the past, until I did a nagios reload.

There was one directive which solved the stale re-check times:-
check_for_orphaned_services=1

Also, I reduced a couple of timeout values so that Nagios stopped wasting
time on checks which were bound to fail:-
service_check_timeout=30
host_check_timeout=30
event_handler_timeout=30
notification_timeout=30

Given that the load on the machine doesn't appear to go over 0.50, I've
allowed infinite concurrent services checks now, increased from 400, but
that appears to be making no difference at all. And I left the reaper
frequency at 10 seconds.  So now the checks were being re-scheduled for
times in the future, and the latencies stopped running away quite so
dramatically.

This is the state of things at the moment:-

Active Service Checks: Time Frame Checks Completed
<= 1 minute: 107 (11.6%)
<= 5 minutes: 593 (64.3%)
<= 15 minutes: 922 (100.0%)
<= 1 hour: 922 (100.0%)
Since program start:   922 (100.0%)

 Metric Min. Max. Average
Check Execution Time:   0.06 sec 19.70 sec 0.139 sec
Check Latency: 0.00 sec 17.19 sec 2.164 sec
Percent State Change: 0.00% 0.00% 0.00%


Passive Service Checks: Time Frame Checks Completed
<= 1 minute: 0 (0.0%)
<= 5 minutes: 0 (0.0%)
<= 15 minutes: 0 (0.0%)
<= 1 hour: 0 (0.0%)
Since program start:   0 (0.0%)

 Metric Min. Max. Average
Percent State Change:   0.00% 0.00% 0.00%


Active Host Checks: Time Frame Checks Completed
<= 1 minute: 1 (0.9%)
<= 5 minutes: 4 (3.6%)
<= 15 minutes: 5 (4.5%)
<= 1 hour: 5 (4.5%)
Since program start:   11 (9.8%)

 Metric Min. Max. Average
Check Execution Time:   0.02 sec 13.52 sec 0.170 sec
Check Latency: 0.00 sec 8.16 sec 0.073 sec
Percent State Change: 0.00% 0.00% 0.00%


Passive Host Checks: Time Frame Checks Completed
<= 1 minute: 0 (0.0%)
<= 5 minutes: 0 (0.0%)
<= 15 minutes: 0 (0.0%)
<= 1 hour: 0 (0.0%)
Since program start:   0 (0.0%)

 Metric Min. Max. Average
Percent State Change:   0.00% 0.00% 0.00%

However, the latencies are creeping upwards again, albeit very very slowly
and at some point I think I'll have to do a reload just to get the checking
back on track again.

Has anyone got any ideas on where I should be looking to make this better?

K
--
Kate Harris
http://www.totkat.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.monitoring-lists.org/archive/users/attachments/20060307/62688950/attachment.html>