Really slow log2ndo

[Mikael Fridh]

On Wed, Sep 17, 2008 at 7:46 PM, Andreas Ericsson <ae at op5.se> wrote:
> Benjamin Krein wrote:
>> Is there a reason log entries aren't split up into multiple fields in
>> the NDO DB?  It seems kind of silly to put the entire log line in a
>> single field when there are very clear delimiters in the line.

These are a few excerpts of the logentries;

Auto-save of retention data completed successfully.
LOG ROTATION: DAILY
CURRENT HOST STATE: bernicla2;UP;HARD;1;PING OK - Packet loss = 0%,
RTA = 0.50 ms
ndomod: Successfully reconnected to data sink!  0 items lost, 88
queued items to flush.
CURRENT SERVICE STATE: rt-130-238-128-160-27;PING;OK;HARD;1;PING OK -
Packet loss = 0%, RTA = 2.77 ms

There are only clear delimiters for some types of entries, not all.
And if you do separate the types into different tables it will stop
being a Log... A log is where you go to check historically what
happened during a certain period of time. If you split it up in
separate tables it will be more complex to get an excerpt on just that
- all events during a certain period.

> I imagine the current structure was designed to facilitate displaying
> the entire log though. Your guess is as good as mine, as it was a long
> time ago I took a look at the ndoutils code.

It's a log on disk, thus it's a log in the database. Atleast that's my
take on it.
I would use MyISAM MERGE tables or PARTITIONS for this if you want to
keep an "online" archive of the logs.
With the MERGE trick you can compress old tables and rejoin them in
the merge periodically if you'd like.

>> I'm contemplating writing my own parser for the archived logs, but I'm
>> tempted to modify the NDO code to make use of multiple fields.

You could modify the ndo code to recognize more event types. There are
a few logentry_type IDs that are duplicate it seems.

The event history of objects (services and hosts) is already in
nagios_statehistory, so what else is there in the log that you could
gain so much from parsing out into separate tables/fields?

--
Mike

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/