Difference in CPU time with and without ePN

[Andreas Ericsson]

Thomas Guyot-Sionnest wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Andreas Ericsson wrote:
>> Thomas Guyot-Sionnest wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> On 09/01/08 03:35 AM, Andreas Ericsson wrote:
>>>> Have you found a way around the memory leakage? Otherwise, I still believe
>>>> it's  more hassle than it's worth, and effort would be better spent to cut
>>>> the number of fork()'s in half by having Nagios multiplex its checks.
>>> I never noticed any memory problem with the ePN and my Nagios often ran
>>> for many consecutive months without being stopped (doing SIGHUPs from
>>> time to time to update the config trough)
>>>
>>> Could you direct me to some documents of communication archives that
>>> point out the problem?
>>>
>> http://www.google.se/search?q=%2Bnagios+%2B%22embedded+perl%22+%2B%22memory+leak%22
>>
>> Embedded perl leaks memory. Alot. If you have a setup where it doesn't,
>> you're pretty much unique. Look for "memory leak" or "embedded perl" in
>> the nagios-devel and nagios-users archives, apart from the link above.
>>
>> Which versions of Nagios and Perl are you using? What system/hw is this
>> on? ld, glibc and gcc versions might also be interesting, as well as
>> which options you used when compiling Nagios.
>>
>> If the plugins are custom ones, that could also be worth having a look
>> at. In so far as I know though, Stanley Hopcroft has been trying well
>> over a year to consign the leaks into oblivion, with some but far from
>> complete success, and the result varies heavily depending on a lot of
>> different things, all of which aren't 100% clear to anyone.
>>
> 
> Well, yesterday I had to kill and restart Nagios to make changes to Perl
> modules apply (HUP wouldn't do) and it's true that Nagios now use much
> less memory, so indeed there seems to be leaks. However with 2GiB of RAM
> it would take months (maybe more than a year) to fill up the server
> memory. I don't graph real memory usage yet (used memory minus
> cache/buffers) so I can't really tell how fast it increase, but it's for
> sure not a big deal with 2GiB.
> 

That depends on the system. Some have reported leaks of 20MiB/minute.
2 gigs won't last very long at that rate.

> HUPs have been said to cause memory leaks as well and
> 1. In average I probably HUP Nagios 2-3 times a week.
> 2. Automated scripts probably do the same on config pulls from AD (The
> HUP only happens if the config changes, but that happens quite often)
> 
> On the solutions side, you can.
> 
> 1. Kill and restart Nagios instead of HUP'ing it (Why don't nagios
> execve itself on HUPs BTW?)

I have no idea. It would definitely make it easier to write modules
for it, as each new load is 100% certain to provide a clean slate.

> 2. Monitor the Nagios process memory usage, with optionally an event
> handler for automated restarts

That feels decidedly icky though. It would be better to use the malloc
info from glibc to internally decide if we're leaking or not, and then
simply execve() self when the memory usage grows too large.

> 3. Schedule automated restarts every day/week/month or so
> 4. Add more memory
> 

Adding more memory only puts off having to do one of the other
things though. It's not a real solution.

> For me the benefits is definitely worth the downsides.
> 

You're not alone. Just be aware that some systems see enormous
leaks when embedded perl is used. I was curious to know what
your specs were, as you would definitely have noticed if you
were suffering from the huge kind of leaks.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace