Children "unreachable" on soft down?

[Israel Brewster]

So is this just something I'll have to live with? I don't seem to be  
getting much feedback on the subject. :(
-----------------------------------------------
Israel Brewster
Computer Support Technician II
Frontier Flying Service Inc.
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7250 x293
-----------------------------------------------



On Apr 6, 2009, at 10:55 AM, Israel Brewster wrote:

> On Apr 6, 2009, at 9:03 AM, Giorgio Zarrelli wrote:
>
>> Hi,
>>
>> I've not quite clear what happens to you,
>
> Thanks for the response. For clarification, the exact sequence of
> events is as follows:
>
> 1) The link between the nagios box and one of our routers, which we
> will refer to as the parent host glitches for 30 seconds or so. Due to
> the nature of the link (satellite connection) this is semi-expected,
> and happens a couple of times a day.
>
> 2) Nagios catches this glitch in one of it's regularly scheduled host
> checks, and puts the parent host into a soft down state. Again, normal
> and expected - even good.
>
> 3) At the same time, Nagios puts the children of the parent host into
> an "unreachable" state. Makes sense, at least, but leads to the issue
>
> 4) The parent host is now in recheck mode (as it is only in a soft
> down state and has three rechecks set), so it checks again a minute
> later. This check succeeds, as the outage was transitory. The parent
> host is put back into an "UP" state. As it never was in a hard "down"
> state, no notification is sent. This is good.
>
> 5) since the parent is now up, the child host now is changed to a
> (soft I think) "down" state.
>
> 6) check continue on a normal schedule. As the link does not glitch
> again for several hours, parent remains up and child remains
> (correctly) down. Three checks later, child enters a hard "down" state
> (since it was unreachable and only just switched back to down). Down
> notification is sent for child.
>
> 7) Everything remains good for the next several hours until the link
> glitches again. Repeat from step one.
>
> The notification in step 6 is the problem here - the child host was
> down before the glitch, the child host is still down after. But
> because the child host was temporarily put in an unreachable state, we
> get notified again that it is down, resulting in a string of "DOWN"
> messages with no up or real change in status.
>
>> but one thing I have in mind is try
>>
>> soft_state_dependencies=0
>>
>> Besides that, the problems seems to be in the roots of the check.
>> It's not
>> healty to have a ping check failing every 2 strikes. Try to change
>> the host
>> alive check, using a ssh check instead.
>
> The check is not failing every 2 strikes. It's failing once, briefly,
> every few hours - just barely long enough to make one check fail and
> throw the parent host into a soft down state. The first recheck (one
> minute later) works fine, bringing the parent back to an up state. The
> next several hundred or more checks also work fine (as the problem was
> transitory and brief). For this reason, changing the check wouldn't
> help - for the duration of that single check, the host really is down
> (or more precisely, unreachable, as it is a link issue), and any check
> I used would say so.
>
>> Another approach, not so useful, would be to increase the timeout
>> for the ping
>> (-W) so it will have less chances to fail.
>
> except that it's not a timeout issue. It is a very real, albeit brief
> (around 30 seconds or so), outage. Not long enough or frequent enough
> to really impact productivity or anything, but long enough for nagios
> to catch it (for a single check).
>
> -----------------------------------------------
> Israel Brewster
> Computer Support Technician II
> Frontier Flying Service Inc.
> 5245 Airport Industrial Rd
> Fairbanks, AK 99709
> (907) 450-7250 x293
> -----------------------------------------------
>>
>> Giorgio
>>
>> Israel Brewster (israel at frontierflying.com) scritto:
>>>
>>> So does anyone have any ideas as to how I can resolve this  
>>> situation?
>>> It continues to be an annoyance. Thanks.
>>>
>>> -----------------------------------------------
>>> Israel Brewster
>>> Computer Support Technician II
>>> Frontier Flying Service Inc.
>>> 5245 Airport Industrial Rd
>>> Fairbanks, AK 99709
>>> (907) 450-7250 x293
>>> -----------------------------------------------
>>>
>>>
>>>
>>> On Mar 31, 2009, at 8:17 AM, Israel Brewster wrote:
>>>
>>>> On Mar 31, 2009, at 1:09 AM, Andreas Ericsson wrote:
>>>>
>>>>> Israel Brewster wrote:
>>>>>> Does nagios (3.0.3) mark a child host as unreachable when its
>>>>>> parent  enters a soft down state? I am finding myself getting
>>>>>> repeated down  messages for a host (which is, in fact, down),  
>>>>>> even
>>>>>> though I have  notifications set to only send a single message.
>>>>>> Looking at the logs,  it would appear that what is happening is
>>>>>> that the host is flipping  between "down" (which notifies me) and
>>>>>> "unreachable" (which does not).  The parent host, however, never
>>>>>> enters a hard down state. Looking at  the logs, what I see is  
>>>>>> that
>>>>>> one ICMP check fails, throwing the host  into a soft down state,
>>>>>> but the next one works just fine, bringing it  back to an up
>>>>>> state.
>>>>>> The logic works fine for the parent host- since it never hits a
>>>>>> hard  down state, it doesn't alert, and everyone is happy. But
>>>>>> apparently  with the child host every time this happens, it
>>>>>> switches from critical  to unreachable and back again,
>>>>>> triggering a
>>>>>> notification. Is there any  way to keep this from happening?
>>>>>> Thanks.
>>>>>
>>>>> Doesn't flapping detection do what you want? You'd get a few
>>>>> notifications, but they'd stop after the 3rd flip or something, I
>>>>> think.
>>>>
>>>> Flapping detection helps, but doesn't solve. For one thing, as you
>>>> mentioned, you still get at least a couple of notifications before
>>>> it
>>>> kicks in. For another thing, this happens with a frequency of
>>>> something like once an hour or so (not consistently), so the host
>>>> will
>>>> flip from down to unreachable and back again, triggering an e-mail,
>>>> perhaps do it a second time, and then it will sit in the correct
>>>> "down" state for the next 50 checks or so (thus canceling any
>>>> flapping
>>>> detection) before repeating the process. It's not like I'm getting
>>>> messages every five minutes or anything, it's just that I'm getting
>>>> repeated down messages every hour or two for hosts that have been
>>>> down
>>>> and haven't actually changed state.
>>>>
>>>> I could, of course, schedule down time, except that I want to be
>>>> notified if/when the people in the remote station get their act
>>>> together and get the machine(s) in question back online. Also that
>>>> is
>>>> only partially effective for machines that have been sent in for
>>>> repair, because I don't really know when the scheduled down time
>>>> will
>>>> be over. They are down, I know they are down, I just don't want to
>>>> be
>>>> told about it every few hours :-)
>>>>
>>>> -----------------------------------------------
>>>> Israel Brewster
>>>> Computer Support Technician II
>>>> Frontier Flying Service Inc.
>>>> 5245 Airport Industrial Rd
>>>> Fairbanks, AK 99709
>>>> (907) 450-7250 x293
>>>> -----------------------------------------------
>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Andreas Ericsson                   andreas.ericsson at op5.se
>>>>> OP5 AB                             www.op5.se
>>>>> Tel: +46 8-230225                  Fax: +46 8-230231
>>>>>
>>>>> Considering the successes of the wars on alcohol, poverty, drugs
>>>>> and
>>>>> terror, I think we should give some serious thought to declaring
>>>>> war
>>>>> on peace.
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Nagios-users mailing list
>>>> Nagios-users at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/nagios-users
>>>> ::: Please include Nagios version, plugin version (-v) and OS when
>>>> reporting any issue.
>>>> ::: Messages without supporting info will risk being sent to /dev/
>>>> null
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Nagios-users mailing list
>>> Nagios-users at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nagios-users
>>> ::: Please include Nagios version, plugin version (-v) and OS when
>>> reporting any issue.
>>> ::: Messages without supporting info will risk being sent to /dev/
>>> null
>>>
>>
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by:
> High Quality Requirements in a Collaborative Environment.
> Download a free trial of Rational Requirements Composer Now!
> http://p.sf.net/sfu/www-ibm-com
> _______________________________________________
> Nagios-users mailing list
> Nagios-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-users
> ::: Please include Nagios version, plugin version (-v) and OS when  
> reporting any issue.
> ::: Messages without supporting info will risk being sent to /dev/null


------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null