I would stop doing the time-consuming ticket integration (or whatever is taking up the time) from within the notification command itself. I don't think you need to build a separate notification server - after all, notification logic is one of the things Nagios does best - but I do think you could build a quick program to do the heavy lifting for notifications, per Nagios' instructions. Without knowing what kind of info you need, I would probably have the Nagios notification command write to a spool directory, then send a signal to a separate program to have it read the spool directory and create/search for tickets etc. <div class="gmail_quote">On Tue, Jan 26, 2010 at 2:02 AM, Mike Lindsey <<a href="mailto:mike-nagios@5dninja.net">mike-nagios@5dninja.net</a>> wrote: <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> A typical first tier notification goes to 20 people. One of those will be a pager, and is very simple. The rest are fairly complex. Notifications include a link to existing and recent tickets in our ticketing system (this also allows me to not send a ticket opening notification if a ticket already exists).. I populate the notification with links to cacti graphs, links to wiki documentation for the event as well as fire off a secondary notification handler that adds in additional information based on the host, service, and state. The first notification of the cycles does all the heavy lifting and takes about 6 seconds. The other 19 finish relatively quickly. I've been thinking of building a notification server - so I could have separate and discrete notification escalations for different service states - which would also let me fire off one notification with just the contents of $ENV{NAGIOS_*}.. Perhaps that's my best option? Martin Melin wrote: <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div class="im"> What kind of notifications are you doing and how many are you sending out? Why does a notification cycle take 9 seconds to complete? </div><div class="im"> On Sat, Jan 23, 2010 at 12:13 AM, Mike Lindsey <<a href="mailto:mike-nagios@5dninja.net" target="_blank">mike-nagios@5dninja.net</a> <mailto:<a href="mailto:mike-nagios@5dninja.net" target="_blank">mike-nagios@5dninja.net</a>>> wrote: What kind of options does one have, if your master nagios server is getting overloaded? I have half a dozen slaves doing polling, submitting passive check results back via send_nsca. The master does no active polling, just event processing, notifications, and web ui. Under normal circumstances, it works alright. But after a restart it can take up to half an hour before the master catches up; and if there are a lot of events, the act of sending out notifications can cause it to fall behind. I'm pre-caching my object file, I'm skipping circular dependency checks, and I've gotten a notification cycle down to 9 seconds. I tried modifying nagios to fork before notifications, but that failed pretty spectacularly; so that 9 seconds is a time where 900 or so passive check submissions block until the notifications are done. Are there any options for running a dual-master setup, or other ways to spread the load across multiple machines? Has anyone patched nsca to submit check results into the checkresults directory, instead of via the nagios.cmd pipe? What kind of improvement can one expect from that? Any other advice? </div></blockquote> -- Mike Lindsey </blockquote></div> -- Martin Melin ____________________________ op5 AB <a href="http://www.op5.com">http://www.op5.com</a> <a href="http://www.op5.org/">http://www.op5.org/</a> <a href="http://www.op5.com/op5/products/network-monitor/nagios/">http://www.op5.com/op5/products/network-monitor/nagios/</a>