Really slow log2ndo

[Benjamin Krein]

Is there a reason log entries aren't split up into multiple fields in  
the NDO DB?  It seems kind of silly to put the entire log line in a  
single field when there are very clear delimiters in the line.

I'm contemplating writing my own parser for the archived logs, but I'm  
tempted to modify the NDO code to make use of multiple fields.

Benjamin Krein
Systems Administrator
AWeber Communications, Inc.



On Sep 16, 2008, at 12:32 PM, Andreas Ericsson wrote:

> Benjamin Krein wrote:
>> I posted the following in the Users mailing list, but got to thinking
>> that maybe Developer's list was a better place for it.
>>
>> Am I the only one who is experiencing very slow performance importing
>> archived log files into MySQL with log2ndo?  I'm averaging ~1.5 log
>> entries per second being inserted.  This seems slow to me.
>>
>> I've tried dividing the parsing & MySQL service across 2 servers as
>> well as move all the parsing/inserting locally on a remote (to  
>> Nagios)
>> server which is much beefier than our actual Nagios server.  My
>> insertion speed remains fairly consistent.
>>
>> I'm currently using the Unix socket option.  Is this the wrong one?
>> Would TCP be faster (doesn't seem so)?  Should I process the logs to
>> flat text files first then push those to ndo2db?
>>
>> At this rate it's going to take FOREVER!  I have ~10 million archived
>> lines to parse and insert.  Maybe I'm expecting too much?
>>
>
> First thing to try is to disable all indexes on the sql table
> while running the import, and then re-enabling them afterwards.
>
>
> If you're feeling hardcore, you could try first using the
> importer from op5 reports-module and then convert the imported
> table to ndoutils format.
>
> The reports-module importer has done live imports of 800M lines
> of log-data in < 6 minutes. You'll find it at http://www.op5.org/
> under the 'projects' section. We have no snapshots available,
> but the gitweb can generate one for you on-the-fly if you want.
>
> Note that it'll drop quite a lot though, as it only bothers with
> host states, service states, downtime start/stop and program start/
> stop.
>
> YMMV.
>
> -- 
> Andreas Ericsson                   andreas.ericsson at op5.se
> OP5 AB                             www.op5.se
> Tel: +46 8-230225                  Fax: +46 8-230231
>
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's  
> challenge
> Build the coolest Linux based applications with Moblin SDK & win  
> great prizes
> Grand prize is a trip for two to an Open Source event anywhere in  
> the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Nagios-devel mailing list
> Nagios-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-devel


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/