trying to fix problem with excessive latency

[Corey Hickey]

On 2010-05-19 11:47, Corey Hickey wrote:
> On 2010-05-18 18:39, Mathieu Gagné wrote:
>> On 2010-05-18 21:29, Corey Hickey wrote:
>>> Hello,
>>>
>>> I have inherited maintenance of a medium-sized Nagios installation. We
>>> currently have 649 hosts and 5415 services. Our setup works nicely, with
>>> one exception: Nagios falls behind on host/service checks. Our usual
>>> latency once Nagios has been running for a while is about 190-200
>>> seconds. Our Nagios host is reasonably powerful and isn't struggling; it
>>> seems that Nagios itself is limited somehow.
>>
>> Try disabling "enable_environment_macros". It helped use greatly when we
>> disabled this option. Nagios was spending most of its time allocating
>> memory for environment variables it never used. (in our case)
>>
>> Host/Service checks latency dropped from ~20 minutes to 10 seconds I
>> would say.
> 
> I just tried this and latency dropped from 171 seconds to 149 seconds.
> Thanks, that definitely helped. I'm actually surprised that it did,
> since our nagios process doesn't seem to be CPU bound (75-85% idle,
> rarely with any individual core below 40% idle). It might be that I'm
> drawing the wrong conclusion from top, though, since the nagios process
> moves around from core to core.
> 
> Anyway, thanks again, and thanks to everyone for the advice on- and
> off-list. I'm going to try more of your suggestions, and I'll let you
> know how it goes.

It looks like use_large_installation_tweaks did the trick. I could have
sworn I tried that before; likely I set it to 0 instead of 1 or
something foolish like that.

Anyway, I wanted to test the individual components of
large_installation_tweaks to see what helped.
* child_processes_fork_twice=0: latency drops from 149 to 18
* free_child_process_memory=0:  latency drops from 149 to 0.6
* disabling summary macros in environment variables doesn't seem to be
  individually configurable, but I expect that's disabled already by
  enable_environment_macros=0.

So, thanks again to everybody who helped. Latency is now 0.35% of what
it used to be.

-Corey

------------------------------------------------------------------------------

_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null