[naemon-users] naemon Startup very slow

[Kottmann, Jörg]

I think it`s a probem with the precaching function, but i will test further

-----Ursprüngliche Nachricht-----
Von: Sven Nierlein [mailto:Sven.Nierlein at Consol.de] 
Gesendet: Freitag, 2. Juni 2017 09:13
An: Kottmann, Jörg <joerg.kottmann at gkvi.de>; 'naemon-users at monitoring-lists.org' <naemon-users at monitoring-lists.org>
Betreff: Re: AW: AW: [naemon-users] naemon Startup very slow

-T just prints additional timing information to stdout. It doesn't make things faster but it maybe reveals some details on what takes so long.

 Sven

On 02/06/17 8:21, Kottmann, Jörg wrote:
> Hi Sven,
>
> just tested the -T Option and the start takes about 6 min. Started the 
> same config with omd start core, the command takes about 6 min to come back and then it takes about 9 min to load the moduls and start the checks.
>
> Regards
> Jörg
>
> -----Ursprüngliche Nachricht-----
> Von: Sven Nierlein [mailto:Sven.Nierlein at Consol.de]
> Gesendet: Donnerstag, 1. Juni 2017 14:59
> An: naemon-users at monitoring-lists.org
> Cc: Kottmann, Jörg <joerg.kottmann at gkvi.de>
> Betreff: Re: AW: [naemon-users] naemon Startup very slow
>
> Hi Jörg,
>
> Sry, i forgot to set the list on cc. So here we are again...
>
> You could run "naemon -T tmp/naemon/naemon.cfg" directly as site user. But it seems like its the amount of dependencies. You could also try the "-x" switch. Although its deprecated and i have no idea if it changes anything anymore.
>
> Cheers,
>  Sven
>
>
> On 01/06/17 10:56, Kottmann, Jörg wrote:
>> Hi Sven,
>>
>> this was the highest debug level. We have many Service dependencys 
>> defined, which increases the startup time, but i think 15 min are too long.
>>
>> If I trace the process, I only see brk requests like the follow
>> brk(0x907c000)                          = 0x907c000
>>
>> This takes ca. 5 min, after then the naemon service is started ( omd start core comes back with ok ). 
>> And then it takes about 10 min till naemon starts loading the modules or checking services.
>> This trace of the prozess doesn`t output anything and the CPU use is 
>> 99%
>>
>> If tried it without the moduls, with no change.
>> The Server has 12 CPUs and 18 GB RAM
>>
>> Regards,
>> Jörg
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Sven Nierlein [mailto:Sven.Nierlein at Consol.de]
>> Gesendet: Donnerstag, 1. Juni 2017 10:34
>> An: Kottmann, Jörg <joerg.kottmann at gkvi.de>
>> Betreff: Re: [naemon-users] naemon Startup very slow
>>
>> Hi Jörg,
>>
>> does it start normal without the modules? Could you increase the debug level even more to see where it takes too long?
>> Gearman and Livestatus are quite common combination and i haven't experienced such issues anywhere yet.
>>
>> Cheers,
>>  Sven
>>
>>
>> On 01/06/17 10:15, Kottmann, Jörg wrote:
>>> Hi,
>>>
>>>  
>>>
>>> I’m testing naemon 1.0.6 under omd-lab, but naemon startup takes about 15 min. Any suggestions why ?
>>>
>>>  
>>>
>>> Here is the debug log:
>>>
>>> [1496228876.626186] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.626228] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.626236] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.626241] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.626252] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628226] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628267] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628302] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628334] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628361] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628384] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628408] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628434] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628467] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628492] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628516] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628548] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628620] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628682] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628815] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628885] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.628926] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496228876.629145] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496229774.828030] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496229774.828066] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496229774.828074] [064.0] [pid=22475] Module '/omd/sites/prod/lib/mod_gearman/mod_gearman_naemon.o' loaded with return code of '0'
>>>
>>> [1496229774.828078] [064.0] [pid=22475] nebmodule_deinit() found
>>>
>>> [1496229774.829615] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496229774.833062] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496229774.833069] [064.2] [pid=22475] Callback #1 (type 2) return 
>>> code = 0
>>>
>>> [1496229774.833077] [064.1] [pid=22475] Making callbacks (type 2)...
>>>
>>> [1496229774.833081] [064.2] [pid=22475] Callback #1 (type 2) return 
>>> code = 0
>>>
>>> [1496229774.833085] [064.0] [pid=22475] Module '/omd/sites/prod/lib/naemon/livestatus.o' loaded with return code of '0'
>>>
>>>  
>>>
>>> Between the start and the load of the last modul is a time difference of 15 minutes.
>>>
>>>  
>>>
>>> Regards,
>>>
>>> Joerg
>>>
>>>  
>>>
>


-- 
Sven Nierlein             Sven.Nierlein at consol.de
ConSol* GmbH              http://www.consol.de
Franziskanerstrasse 38    Tel.:089/45841-439
81669 Muenchen            Fax.:089/45841-111