So I found an obscure "su hangs" message board posting that recommended restarting syslogd. I am running syslog-ng, and after restarting the daemon I was able to start Nagios without any problems. Local 'su - nagios' commands also work without any delay. I suspect that some interaction between Nagios and syslog-ng is causing Nagios to stop working, and the subsequent restart fails as a result of that original problem.<br>
<br>I'll probably move this problem to the syslog-ng mailing list, but has anyone ever seen this before? At the very least maybe this will provide someone who has this problem in the future with more information.<br clear="all">
<br>-- Eric Cables<br>
<br><br><div class="gmail_quote">On Thu, Jul 1, 2010 at 2:36 PM, Eric Cables <span dir="ltr"><<a href="mailto:ecables@gmail.com">ecables@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Well, I tried to duplicate the command that is showing up in the 'ps -xw' output, and it just hangs.<br><br>[nagios@psdbsd01 (~)]$ whoami<br>nagios<br>[nagios@psdbsd01 (~)]$ su - nagios -c touch /usr/local/nagios/var/nagios.log /usr/local/nagios/var/retention.dat<br>
<br>^^ hangs here.<br><br>In fact, if I just try to 'su - nagios' the process hangs as well. <br><br>Using su with other parameters works, however, so the binary seems to function:<br>[nagios@psdbsd01 (~)]$ su -<br>
Password:<br>[root@psdbsd01 (~)]# <br><br>And su - nagios from the root user appears to work fine.<br>[root@psdbsd01 (~)]# su - nagios<br>[nagios@psdbsd01 (~)]$ <br><br>But su - nagios does not (as the nagios user):<br>[nagios@psdbsd01 (~)]$ su - nagios<br>
<br>^^ hangs<br><br>Sorry for all the noise.<br><font color="#888888"><br>-- Eric Cables</font><div><div></div><div class="h5"><br>
<br><br><div class="gmail_quote">On Thu, Jul 1, 2010 at 2:15 PM, Eric Cables <span dir="ltr"><<a href="mailto:ecables@gmail.com" target="_blank">ecables@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Here are a few more details I've been able to gather.<br><br>Here's the output of a truss on the init script w/ the start statement:<br>Starting nagios:write(1,"Starting nagios:",16) = 16 (0x10)<br>
fork(0x90,0xbfbfe9f8,0xa,0x8062a35,0x0,0x0) = 55445 (0xd895)<br>getpgrp(0x0,0x0,0xd895,0x0,0x2831c0c0,0x0) = 55444 (0xd894)<br>wait4(0xffffffff,0xbfbfe9d8,0x2,0x0,0x213,0x1) = 55445 (0xd895)<br>stat("/sbin/su",0xbfbfe6f8) ERR#2 'No such file or directory'<br>
stat("/bin/su",0xbfbfe6f8) ERR#2 'No such file or directory'<br>stat("/usr/sbin/su",0xbfbfe6f8) ERR#2 'No such file or directory'<br>stat("/usr/bin/su",{ mode=-r-sr-xr-x ,inode=14512669,size=14496,blksize=4096 }) = 0 (0x0)<br>
fork(0x0,0x0,0x4b156e10,0x0,0x0,0x0) = 55446 (0xd896)<br>getpgrp(0x0,0x0,0xd896,0x0,0x2831c0c0,0x0) = 55444 (0xd894)<br><br>^^^ This is where it hangs.<br><br>ps -ax | grep nagios shows the following:<br>
55443 6 I+ 0:00.02 truss /usr/local/etc/rc.d/nagios.sh start<br>55444 6 IX 0:00.01 /bin/sh /usr/local/etc/rc.d/nagios.sh start<br>55447 6 S 0:00.07 su - nagios -c touch /usr/local/nagios/var/nagios.log /usr/local/nagios/var/retention.dat<br>
<br>Here is retention.dat (not sure why it would hang here):<br>-rw------- 1 nagios nagios 2008435 Jul 1 12:26 retention.dat<br><br>These are really the only clues I'm able to find at this point.<br><font color="#888888"><br>
-- Eric Cables</font><div><div></div><div><br>
<br><br><div class="gmail_quote">On Thu, Jul 1, 2010 at 2:09 PM, Eric Cables <span dir="ltr"><<a href="mailto:ecables@gmail.com" target="_blank">ecables@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Thanks for the reply. I ended up rebooting the box, which fixed the problem temporarily, but it has resurfaced again. When I drill down into a service check it says that the next check will be processed at a time that has already passed.<br>
<br>For example:<br>Last Check: 13:09<br>Next Check: 13:11<br><br>The current time on, however, is 14:02...<br><br>When I try to stop the process via the init script I get the following:<br>[nagios@psdbsd01 (~/var)]$ /usr/local/etc/rc.d/nagios.sh stop<br>
Stopping nagios: ..........<br>Warning - nagios did not exit in a timely manner<br><br>The cmd file does not exist prior to attempting to start, after stopping, but I back to the problem where Nagios will not start and instead hangs indefenitely when requested to start. <br>
<br>[nagios@psdbsd01 (~/var)]$ /usr/local/etc/rc.d/nagios.sh start<br>
Starting nagios: <-- hangs here<br><br>I'm not sure about the lock file, this is a FreeBSD install from source, and I don't see a /var/lock directory at all. Everything Nagios related is installed in /usr/local/nagios as far as I can tell.<br>
<br>There doesn't seem to be anything of interest in nagios.log, as the last entry just reports a notification that was sent out prior to Nagios losing its functionality.<br><br>Any other tips? I'm not exactly sure why a reboot fixed this before, but any speculation is appreciated.<br clear="all">
<font color="#888888">
<br>-- Eric Cables</font><div><div></div><div><br>
<br><br><div class="gmail_quote">On Thu, Jul 1, 2010 at 6:05 AM, Jim Avery <span dir="ltr"><<a href="mailto:jim@jimavery.me.uk" target="_blank">jim@jimavery.me.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div>On 1 July 2010 01:18, Eric Cables <<a href="mailto:ecables@gmail.com" target="_blank">ecables@gmail.com</a>> wrote:<br>
> Sorry to bug the list, but my 3.2.1 installation of Nagios has all of a<br>
> sudden stopped starting. I noticed a lack of alerts over the last day, and<br>
> when I checked the GUI it indicated that the "next" scheduled check for a<br>
> service was in the past. I proceeded to stop/start Nagios, but both have<br>
> failed.<br>
><br>
> Currently when I try to start Nagios using the init script it just hangs:<br>
> [nagios@psdbsd01 (~/etc)]$ /usr/local/etc/rc.d/nagios.sh start<br>
> Starting nagios:<br>
><br>
> I've enabled debug logging (-1 level, 2 verbosity), but this is all that<br>
> shows up in nagios.debug when I issue the above start request (uid 1003 =<br>
> nagios):<br>
> [1277942532.270096] [001.0] [pid=46503] drop_privileges() start<br>
> [1277942532.270194] [004.0] [pid=46503] Original UID/GID: 1003/1003<br>
><br>
> I can run nagios -v nagios.cfg, and it reports no errors.<br>
><br>
> Here's the output if I run nagios nagios.cfg manually, without invoking<br>
> daemon mode:<br>
> [nagios@psdbsd01 (~/etc)]$ ../bin/nagios ./nagios.cfg<br>
><br>
> Nagios Core 3.2.1<br>
> Copyright (c) 2009-2010 Nagios Core Development Team and Community<br>
> Contributors<br>
> Copyright (c) 1999-2009 Ethan Galstad<br>
> Last Modified: 03-09-2010<br>
> License: GPL<br>
><br>
> Website: <a href="http://www.nagios.org" target="_blank">http://www.nagios.org</a><br>
><br>
> Any tips? I am not sure what the next steps are since both logging and<br>
> debugging aren't producing output, and Nagios has never taken more than a<br>
> few seconds to start in the past.<br>
<br>
</div></div>What, if anything, shows up in your nagios.log file?<br>
<br>
Check you don't already have a nagios daemon running (ps -ef | grep<br>
nagios) before you start it again.<br>
<br>
Check that the lock file isn't there from the previous invocation (if<br>
you did a standard install from source tarballs the file is<br>
/var/lock/subsys/nagios).<br>
<br>
Check that the Nagios command file /usr/local/nagios/var/rw/nagios.cmd<br>
doesn't exist before you start nagios.<br>
<br>
Use full pathnames when attempting to verify your config, for example:<br>
<br>
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg<br>
<div><div></div><div><br>
------------------------------------------------------------------------------<br>
This SF.net email is sponsored by Sprint<br>
What will you do first with EVO, the first 4G phone?<br>
Visit <a href="http://sprint.com/first" target="_blank">sprint.com/first</a> -- <a href="http://p.sf.net/sfu/sprint-com-first" target="_blank">http://p.sf.net/sfu/sprint-com-first</a><br>
_______________________________________________<br>
Nagios-users mailing list<br>
<a href="mailto:Nagios-users@lists.sourceforge.net" target="_blank">Nagios-users@lists.sourceforge.net</a><br>
<a href="https://lists.sourceforge.net/lists/listinfo/nagios-users" target="_blank">https://lists.sourceforge.net/lists/listinfo/nagios-users</a><br>
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.<br>
::: Messages without supporting info will risk being sent to /dev/null<br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>