Limiting absolute floods of emails/sms
Andreas Ericsson
ae at op5.se
Thu Oct 20 18:46:02 CEST 2005
Nathan Oyler wrote:
>>Nathan Oyler wrote:
>>
>>>I wanted to know if anyone had some tips in limiting floods of
>
> emails or
>
>>>sms messages.
>>>
>>>Right now we use parent child relationships, but with 400 hosts
>
> there
>
>>>are times when the oddest things happen which will cause a huge
>
> amount
>
>>>of flooding.
>>>
>>>Is there a way to cap the amount of notifications that come out at a
>>>time?
>>>
>>
>>Not really, except disabling notifications when the shit hits the fan.
>>If you're talking about notifications that nobody cares about you need
>>to tweak your thresholds, notification_options or
>
> notification_periods.
>
> It's notifications that people care about, it's really the amount of
> them.
>
> Example, today we've had an issue where several machines from one
> location to another have experienced packet losses.
>
> Each one causes a warning or critical notification, and a recovery.
> That's a lot of messages while we work on correcting the problem at one
> time. I could schedule downtime for that location but that is a pretty
> harsh reaction to the problem.
>
> It would be preferable to do something like catch all messages every 15
> minutes, and then a condensed version of what's going on. One
> notification email which lists each issue for the last 15 minutes? Maybe
> if there have been 5 emails or more in the last 15 minutes, or some
> other criteria.
>
> To do something like this, would one want to edit the command for emails
> to go out, and change it to call a script which would hold the
> information as to what was happening, and deal with it in that respect.
>
Yes.
> Would this be a bad idea in your mind?
>
That depends on the implementation. A neat and fairly nice way of doing
it would be to shoot them into a daemon which could keep track of the
number of notifications for each user. When it exceeds a given threshold
it could put them in a database of some sort and wait a given amount of
time before sending any more. Attached to the database you could
ofcourse have a small php-script to view and delete pending
notifications before they're sent. It would also be quite cool if the
little daemon could notify you (once) of the fact that it's blocking
notifications for the pre-determined period of time before sending them on.
Sounds like a fun project. Let me know if you need any contract work
done. I'll get you a good pricetag on it.
--
Andreas Ericsson andreas.ericsson at op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue.
::: Messages without supporting info will risk being sent to /dev/null
More information about the Users
mailing list