Load issues with Nagios

[Ken Snider]

Greetings all,

We have Nagios running, about 50 servers, 250 processes monitored. The vast 
majority of these checks are via nsca/send_nsca, with the only active checks 
being host-checks, and ping checks. nsca is running via Xinetd, 
command_check_interval is -1, reaper frequency is 5, and everything works 
reasonably well. Services are configured to check in every 2 minutes.

My issue is, the box is *always* at 100% CPU. At first, I figured this was 
related to the 50 or so send_nsca connections (and subsequent dumps to 
Nagios' pipe) that occur every two minutes via nsca. However these are dealt 
with within 20 or so seconds, leaving about a minute and a half where all 
Nagios is really doing is pinging.. yet the CPU usage remains.

My next thought was the command_check_interval being -1. setting it to 1 had 
no difference.

I tried raising the reaper interval to 10 seconds as well, no difference. I 
lowered the max_concurrent_checks to 40, no change. consistently 100% use.

Set the status_update interval to 30 seconds, to make sure it wasn't the 
writes to the status file, nothing (yep, aggregate writes is on).

The system, BTW, is RH8, PIII-750, with 256 MB RAM and a Gig of Swap. The 
box is using a whopping 60MB of ram. The box almost never uses IO (save the 
mad rush every two minutes from NSCA), and has nothing else of import (or 
load) running. Load is all from the parent nagios process.

So, here's my question.. is this load, perhaps, *normal*? Is a P-III 750 
really, truly maxed out with about 50 active services (ping mainly), and 300 
passive services running?

I personally find that difficult to believe, but I like to hear everyone's 
thoughts on the subject. ;)

Thanks in advance. :)

-- 
Ken Snider
Senior Systems Administrator
Datawire Communication Networks Inc.



-------------------------------------------------------
This SF.NET email is sponsored by:
SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
http://www.vasoftware.com