"(No output!)" Critical Alert Solution

[ffejes at sears.com]

Hello all.  A few weeks ago I sent out an email regarding plugin alerts
being created with "(No output!)" as the alert output.  Nobody responded to
the message but, nonetheless, I have found a solution and would like to
share in the event that anybody ever comes across a similar situation.
Here is some background information:

+ We've been running NetSaint/Nagios for over 6 months.
+ We currently have 250 hosts being monitored and ~300 services.
+ The server is running NetBSD/i386 1.5.2.
+ We began seeing these alerts after growing to be a little more than 200
hosts.

After a lot of unsuccessful attempts at modifying the actual Nagios code
and getting nowhere (and even trying to downgrade Nagios), I decided that
the problem had to be due to the increased number of managed hosts.  The
"(No output!)" message is the default output to a plugin response and it
remains if the plugin terminates unexpectedly.  In a nutshell, I could not
think of many reasons why the program would terminate other than some kind
of resource issue.  A quick look at our default NetBSD ulimits gave me hope
as the stack/file descriptors/processes limits were terribly small.  I
bumped them up in the nagios init script by adding the following:

ulimit -n unlimited
ulimit -s unlimited
ulimit -p unlimited

That solved the problem and we've been running without a hitch for the past
few weeks.  Some preliminary testing made me believe that it was the
stacksize setting that made the difference but I did not go too deep and
could not say for absolute certain at this point.

At any rate, this is probably of limited use to most people as the Linux
distributions seem to have much higher default ulimits, but if it prevents
another person from beating his head against the wall for a week or two
then I'll be pleased. :)

Happy monitoring!

--frank




-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31