Nagios lockup for about 8.5 hours

[Andrew Noonan]

Not that I can determine.  We run everything in Central time, and the
nagios user is currently running in that TZ.  Actually, now that I
think about it, it's a fairly moot point, as the logs that I listed
don't have a timestamp, they are epoch stamps.  Since those show a ~5
hour jump, this could not be a TZ change, but either a pause with
Nagios, or the system clock would have to be changed, as that would
effect the epoch values.  Given the consistency of the 5-minute cron
entries I mentioned earlier, I don't think a system clock change
happened either.

On Thu, Jul 9, 2009 at 8:56 AM, Kevin Keane<subscription at kkeane.com> wrote:
> Has the time zone that Nagios runs under changed, maybe? That would not
> affect the log files or NTP, since both usually always run on UTC.
>
> Andrew Noonan wrote:
>> Sorry Kevin, I was out yesterday or I would have responded earlier.  I
>> don't think that's the case.  I forgot to mention it in the earlier
>> email, but I checked the log files of a periodic cron job that also
>> runs on the same server every 5 minutes, and its logs show an
>> uninterrupted timestamp.  In addition, I also monitor NTP through
>> nagios (and graph with PNP), and up until the outage, the local skew
>> was less then a second.
>>
>> Thanks,
>> Andrew
>>
>> On Tue, Jul 7, 2009 at 9:56 PM, Kevin Keane<subscription at kkeane.com> wrote:
>>
>>> It seems to me that for some reason your system clock has changed by
>>> about five hours. Did you change your system by any chance from local
>>> time (Eastern time, probably, based on the five-hour difference) to UTC?
>>> Or maybe your clock had drifted for a long time. When the clock skew
>>> becomes too great, NTP refuses to update the time (because there is no
>>> way to be sure that the time signal isn't the one that's incorrect). If
>>> you restart NTP, it will set your clock regardless of the clock skew.
>>>
>>> The following "immediate check" messages probably occurred because
>>> Nagios thought that these services hadn't been checked for five hours.
>>>
>>> Andrew Noonan wrote:
>>>
>>>> I've been testing out Nagios in general to replace our current system
>>>> and I noticed a strange blank in my PNP graphs this morning.  When I
>>>> looked closer, I found that nagios had basically hung for several
>>>> hours.  Then, the log shows a warning of:
>>>>
>>>> [1246958195] Warning: A system time change of 0d 4h 56m 48s (forwards
>>>> in time) has been detected.  Compensating...
>>>>
>>>> and then for several hours, messages like:
>>>>
>>>> [1246958830] Warning: The check of host 'superhost1' looks like it was
>>>> orphaned (results never came back).  I'm scheduling an immediate check
>>>> of the host...
>>>>
>>>> I'm running nagios 3.0.6 with ndo2db.  The system has under 1000
>>>> services, most of which are nrpe checks to remote hosts.
>>>>
>>>> The nagios system was not terribly loaded at the time (about 50% idle)
>>>> and mysql did not show any errors at the time.  Typically, the number
>>>> of buffers used is only 2-3 out of the 4096.
>>>>
>>>> Any ideas as to what this could have been, or how I can detect this
>>>> condition or log to gain more info?  I wouldn't think that this is
>>>> normal, but my Google searches aren't turning up a lot.
>>>>
>>>> Thanks!
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Enter the BlackBerry Developer Challenge
>>>> This is your chance to win up to $100,000 in prizes! For a limited time,
>>>> vendors submitting new applications to BlackBerry App World(TM) will have
>>>> the opportunity to enter the BlackBerry Developer Challenge. See full prize
>>>> details at: http://p.sf.net/sfu/blackberry
>>>> _______________________________________________
>>>> Nagios-users mailing list
>>>> Nagios-users at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/nagios-users
>>>> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
>>>> ::: Messages without supporting info will risk being sent to /dev/null
>>>>
>>>>
>>> --
>>> Kevin Keane
>>> Owner
>>> The NetTech
>>> Find the Uncommon: Expert Solutions for a Network You Never Have to Think About
>>>
>>> Office: 866-642-7116
>>> http://www.4nettech.com
>>>
>>> This e-mail and attachments, if any, may contain confidential and/or proprietary information. Please be advised that the unauthorized use or disclosure of the information is strictly prohibited. The information herein is intended only for use by the intended recipient(s) named above. If you have received this transmission in error, please notify the sender immediately and permanently delete the e-mail and any copies, printouts or attachments thereof.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Enter the BlackBerry Developer Challenge
>>> This is your chance to win up to $100,000 in prizes! For a limited time,
>>> vendors submitting new applications to BlackBerry App World(TM) will have
>>> the opportunity to enter the BlackBerry Developer Challenge. See full prize
>>> details at: http://p.sf.net/sfu/Challenge
>>> _______________________________________________
>>> Nagios-users mailing list
>>> Nagios-users at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nagios-users
>>> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
>>> ::: Messages without supporting info will risk being sent to /dev/null
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> Enter the BlackBerry Developer Challenge
>> This is your chance to win up to $100,000 in prizes! For a limited time,
>> vendors submitting new applications to BlackBerry App World(TM) will have
>> the opportunity to enter the BlackBerry Developer Challenge. See full prize
>> details at: http://p.sf.net/sfu/Challenge
>> _______________________________________________
>> Nagios-users mailing list
>> Nagios-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nagios-users
>> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
>> ::: Messages without supporting info will risk being sent to /dev/null
>>
>
>
> --
> Kevin Keane
> Owner
> The NetTech
> Find the Uncommon: Expert Solutions for a Network You Never Have to Think About
>
> Office: 866-642-7116
> http://www.4nettech.com
>
> This e-mail and attachments, if any, may contain confidential and/or proprietary information. Please be advised that the unauthorized use or disclosure of the information is strictly prohibited. The information herein is intended only for use by the intended recipient(s) named above. If you have received this transmission in error, please notify the sender immediately and permanently delete the e-mail and any copies, printouts or attachments thereof.
>
>
> ------------------------------------------------------------------------------
> Enter the BlackBerry Developer Challenge
> This is your chance to win up to $100,000 in prizes! For a limited time,
> vendors submitting new applications to BlackBerry App World(TM) will have
> the opportunity to enter the BlackBerry Developer Challenge. See full prize
> details at: http://p.sf.net/sfu/Challenge
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null
>

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge  
This is your chance to win up to $100,000 in prizes! For a limited time, 
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize  
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null