High latencies problem.

[Alessandro Ren]

On 3/26/2009 3:05 AM, Hendrik Baecker wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Alessandro Ren schrieb:
>    
>>       Just to let you know that my latency gets under 0.2s if a disabled
>> performance data processing.
>>      
> 1) Nice to hear about the nice performance
> 2) Bad to hear that performance data processing was your bottleneck
>
>    
>>       I´ve changed the way we process them and everything is very nice
>> with 11k services.
>>
>>      
> How did you process them before your action?
> That could be interesting for other users who ran into that problem on a
> bigger installation, and maybe we are able to find a better way overall?
>    

     I have nagios writing all perfomance data to a file and rotating it 
each 60s and I´ve a importer written in C, perl would to catch up with 
the files being rotated, to import them each 3 minutes.
     Yesterday we went to 19k services, more then 50% are checs with 
NRPE, a lot of perls beeng run and latency is te 0.4s using the schema 
above.
     I am running nagios on a two node cluster, 64bits RedHat 
enterprise, each node is a Two dual core Xeon 3GHz processors with 8GB 
of RAM.
     Mysql runs on this cluster  in a multi-master replication setup and 
I´ve nagios running on one side with the importer and the CGIs and our 
own custom interface running on the other, with rsync sending the status 
file each 2 seconds two the other side. All watched by heartbeat.
     I´ve noticed that with this many sevices, the interface get very 
slow and starves the server, so I´ve  moved all comments to the 
database, as nagios would go throught the comments linked list for each 
service, changed the nagios code to write it direclty there and the CGis 
to read from there. We also serialize all status data to a memcahe to 
speed up the interface, creating and 70s cache, which worked very nice 
brought the load that was at 15/17 to 2/4.

     []s.

>
> Regards
> Hendrik
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
>
> iEYEARECAAYFAknLGysACgkQlI0PwfxLQjmw9ACeNLUkL7dvtSimIkA2kMSIWEY0
> e+YAniQ6bWoh8JKX7Oez83Gx+ZR1/p1W
> =0eYu
> -----END PGP SIGNATURE-----
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Nagios-devel mailing list
> Nagios-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nagios-devel
>    

------------------------------------------------------------------------------