Consolidating Log Files

[Andreas Ericsson]

Christopher McAtackney wrote:
> Hi all,
> 
> I'm thinking of setting up my Nagios system in a "Redundant
> Monitoring" configuration - two servers (a primary and secondary)
> which both monitor all hosts / services, but only one of which has
> notifications enabled (the primary).
> 
> Upon failure of the primary, the secondary will enable its own
> notifications (and upon the primary returning to an OK state,
> notifications will be disabled).
> 
> One aspect of this that I am wondering how to tackle is consolidating
> my log files after a failure has occurred.
> 
> I want my primary Nagios server's log files to reflect the total
> history of check results that have been made - and so would like the
> results from the secondary server for the time period when the primary
> was done, to be consolidated into the log files of the primary (for
> browsing through the web UI).
> 
> Has anyone looked into this before, and how did you solve the problem
> of duplicate result checks being created in the primary log?
> 

Fortunately, line-based merging of files where every line is prefix
with a timestamp is as simple as doing

cat file1 file2 file3 filen... | sort -n > merged-file

Since the primary server won't be logging at all during the time it's
offline, this merge-strategy will work quite nicely. Do note that it
requires quite a lot of memory and is fairly cpu intensive though, so
if you have a large installation, you'll want to filter out the period
where the primary server was up from the logs of the secondary server
and then just paste those in at the end of the primary server's logs.
Some sed or perl-scripting can help you there.

Note that for either of the above ways to work, the two server's clocks
have to be in sync. If they're not, you need to take the clock-delta
into consideration when merging the logs.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and 
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today. 
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null