System Log monitoring with nagios?

[Jim Mozley]

Jeffery P. Humes wrote:
> What is everyone out there doing for system log monitoring with Nagios.
> 
> I am just looking to monitor the normal logs for regex and report the 
> stuff I want.

A check of the archives should find submissions from myself and (even 
better) Stanley Hopcroft regarding this.

In summary, apart from using the syslog monitoring plugins one can use 
SEC/swatch or similar to watch the log file(s) and then a script to pass 
events of interest into Nagios as passive checks.


 From a previous post (in thread "check_log not working properly"):

The following is a setup for using syslog-ng, swatch and a script to 
pass passive events into Nagios. This is based on getting syslog 
messages from a set of network devices, although the same principle 
would apply to hosts. I've tried to make the example fairly generic, for 
instance we generate the nagios config for this automatically from 
device configurations, so please don't copy/paste this without 
understanding what's going on.

1. Define services in Nagios. For instance if you want to monitor MPLS 
messages for each network device define an MPLS service as below.

#Service definition template
define service{
         name                            mpls-service-template
         active_checks_enabled           no
         passive_checks_enabled          1
         parallelize_check               1
         obsess_over_service             0
         check_freshness                 0
         freshness_threshold             28800
         notifications_enabled           1
         event_handler_enabled           1
         flap_detection_enabled          1
         process_perf_data               1
         retain_status_information       1
         retain_nonstatus_information    1
         register                        0
         is_volatile                     1
         check_period                    none
         max_check_attempts              1
         normal_check_interval           5
         retry_check_interval            1
         contact_groups                  network-admins
         notification_interval           120
         notification_period             24x7
         notification_options            w,c,r
         }

define service{
         use             mpls-service-template         ; template
         host_name       host1
         service_description             mpls
         check_command   ""
         }


2. Configure syslog-ng. Once you have got your config file ready you 
will need to disable the native syslog daemon and start syslog-ng.

syslog-ng config; this is for Solaris the source may need to change for 
linux/BSD:

options {
         keep_hostname(off);
         long_hostnames(off);
         sync(1);
         log_fifo_size(2048);
         bad_hostname("%");
         };

source all {
         sun-stream("/dev/log" door("/etc/.syslog_door"));
         internal();
         udp();
};


# Put each day's log in a separate file within a directory for
# each host.
destination d_hosts {
         file("/var/log/hosts/$HOST/$HOST-$YEAR$MONTH$DAY"
         owner(root) group(syslog) perm(0660) dir_perm(0750) 
create_dirs(yes));
};


# To send messages to swatch
destination d_swatch {
         program("/usr/local/bin/swatch --config-file=/etc/swatchrc 
--read-pipe=\
"cat /dev/fd/0\"");
};


# log all messages in a directory per host
log {
         source(all);
         destination(d_hosts);
};

# send all logs to swatch
log {
         source(all);
         destination(d_swatch);
};



3. Install and configure swatch.

Sample line from swatchrc

watchfor /pattern-i-want-to-match/
         exec /path/to/my/script.pl $*

So for instance if you are looking out for MPLS messages and your 
devices include "MPLS" in these syslog message match the pattern /MPLS/

4. Create the script

This should build a passive command based on the contents of the syslog 
message.

For instance it might contain element such as:

if ( $syslogmsg =~ /down/ ) {
     $nagios_code = 2; # Critical
}

You will need to extract the hostname from the syslog message (and 
possibly the service unless this is hardcoded - it is in this example).

In the end you want to build a message such as:

my $cmd = "[$epoch] 
PROCESS_SERVICE_CHECK_RESULT;$host;$service;$nagios_code;$msg";

And fire it into nagios:

my $echo = '/usr/bin/echo';
my $pipe = '/usr/local/nagios/var/rw/nagios.cmd';

system "$echo \"$cmd\" >> $pipe";

(As an aside I tried the perl way of doing this rather than a system 
command and had a problem on Solaris I couldn't resolve.)

I would recommend you take a look at Al Toby's module on CPAN for 
passing commands to Nagios.

There are also some shell script examples of passing Nagios commands 
supplied with the distribution.


-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null