Difference in CPU time with and without ePN

[Thomas Guyot-Sionnest]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/01/08 04:15 AM, Andreas Ericsson wrote:
> Thomas Guyot-Sionnest wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Well, yesterday I had to kill and restart Nagios to make changes to Perl
>> modules apply (HUP wouldn't do) and it's true that Nagios now use much
>> less memory, so indeed there seems to be leaks. However with 2GiB of RAM
>> it would take months (maybe more than a year) to fill up the server
>> memory. I don't graph real memory usage yet (used memory minus
>> cache/buffers) so I can't really tell how fast it increase, but it's for
>> sure not a big deal with 2GiB.
>>
> 
> That depends on the system. Some have reported leaks of 20MiB/minute.
> 2 gigs won't last very long at that rate.

For now what I can see is that Nagios loose track of roughly 2 MiB per
HUP. That alone can probably explain the memory usage I had over the
last months.

If I look at the virtual size and vss directly from proc (to avoid
rounding) they seems very steady, with a big jump on each HUP:

cat /proc/<pid>/stat|cut -d' ' -f23-24
45871104 6566
[Wait a few minutes]
cat /proc/<pid>/stat|cut -d' ' -f23-24
45871104 6566
[killall -HUP nagios]
cat /proc/<pid>/stat|cut -d' ' -f23-24
46919680 6700
[wait a few minutes again]
cat /proc/<pid>/stat|cut -d' ' -f23-24
46919680 6700


>> HUPs have been said to cause memory leaks as well and
>> 1. In average I probably HUP Nagios 2-3 times a week.
>> 2. Automated scripts probably do the same on config pulls from AD (The
>> HUP only happens if the config changes, but that happens quite often)
>>
>> On the solutions side, you can.
>>
>> 1. Kill and restart Nagios instead of HUP'ing it (Why don't nagios
>> execve itself on HUPs BTW?)
> 
> I have no idea. It would definitely make it easier to write modules
> for it, as each new load is 100% certain to provide a clean slate.

Ethan, are you reading this?

>> 2. Monitor the Nagios process memory usage, with optionally an event
>> handler for automated restarts
> 
> That feels decidedly icky though. It would be better to use the malloc
> info from glibc to internally decide if we're leaking or not, and then
> simply execve() self when the memory usage grows too large.

The event handler would definitely need to do some sanity checking but I
think it's a safe way. Maybe this could be made as a module as well...

>> 3. Schedule automated restarts every day/week/month or so
>> 4. Add more memory
>>
> 
> Adding more memory only puts off having to do one of the other
> things though. It's not a real solution.

That's true for a 20MB/minute kind of leak, but in my case (2 MiB/HUP
leaks) I would definitely have noticed it earlier if I didn't have as
much memory...

>> For me the benefits is definitely worth the downsides.
>>
> 
> You're not alone. Just be aware that some systems see enormous
> leaks when embedded perl is used. I was curious to know what
> your specs were, as you would definitely have noticed if you
> were suffering from the huge kind of leaks.

Was this on Linux? Older Perl release? Was it happening on all Perl
plugins or only on some of them?

Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHhfZm6dZ+Kt5BchYRAgStAKDAkdVEv8O54SzSz0V9wvhNsSghqwCg0I1A
rrJIUAWyRHmVndX1iLxaM9U=
=U2MM
-----END PGP SIGNATURE-----

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace