<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"> <meta name=Generator content="Microsoft Word 12 (filtered medium)"> <style>  </style>  </head> <body bgcolor=white lang=IT link=blue vlink=purple> <div class=Section1> Hi Michael<o:p></o:p> <o:p> </o:p> Thank you very much for the explanations. Finally I found the problem, it was a Nagios issue, too many reload where interrupting data flow.<o:p></o:p> Now everything works very well.<o:p></o:p> By<o:p></o:p> Marco<o:p></o:p> <o:p> </o:p> <div> <div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'> From: Michael Friedrich [mailto:michael.friedrich@univie.ac.at] Sent: domenica 20 settembre 2009 2.34 To: Nagios Developers List Subject: Re: [Nagios-devel] NDO utils bug/explanation<o:p></o:p> </div> </div> <o:p> </o:p> <tt>Hi,</tt> <tt>Frassinelli, Marco wrote:</tt> <o:p></o:p> <tt>Hello</tt><o:p></o:p> <tt>The process of starting ndo2db and then Nagios makes sure that there is actual data within the DB. </tt><tt>If there is an outdated data within the DB it needs to be removed before Nagios even sends new data. So the process of trimming those table entries is truly intentional at the beginning (so-called pre-launch state where the if condition matches). If ndo2db fails for some reason, those data will remain within the database and then removed during the next start. </tt><o:p></o:p> <tt> </tt><tt>Ok, when the parent ndo2db process starts. But not when the child starts.</tt><o:p></o:p> <tt>Considering the fact that the process runs daemonized and forks a child for each ndomod client connection, that is truly wrong. See the NDOUtils documentation page 8:</tt> <tt><a href="http://www.scribd.com/doc/17608352/Nagios-NDOUtils">http://www.scribd.com/doc/17608352/Nagios-NDOUtils</a></tt> <tt>Going deeper into the code in ndo2db.c:main you will recognize that after initialization stuff that the standalone daemon condition matches.</tt> <tt> /* standalone daemon... */</tt> <tt> else{</tt> <tt> /* create socket and wait for clients to connect */</tt> <tt> if(ndo2db_wait_for_connections()==NDO_ERROR)</tt> <tt> return 1;</tt> <tt> }</tt> <tt>This function opens the socket wether tcp or unix. Then the parent forks a child to handle client data (0 = everythings ok, -1 = error).</tt> <tt> /* fork... */</tt> <tt> new_pid=fork();</tt> <tt> switch(new_pid){</tt> <tt>[...]</tt> <tt> case 0:</tt> <tt>#endif</tt> <tt> /* child processes data... */</tt> <tt> ndo2db_handle_client_connection(new_sd);</tt> <tt> /* close socket when we're done */</tt> <tt> close(new_sd);</tt> <tt>#ifndef DEBUG_NDO2DB</tt> <tt> return NDO_OK;</tt> <tt> break;</tt> <tt>[...]</tt> <tt> }</tt> <tt>Parent runs in a while loop and if child has died on the client connection, it forks a new one.</tt> <tt>The child then reads the data from the socket within ndo2db_handle_client_connection. If there's nothing on the socket, the client connection was lost and the while(1) terminates (also on errors), meaning the child to terminate too. The child performs the memory cleanup and db disconnect and then the process of the parent accepting new client connection and forking a child for each client connection starts again. </tt> <tt>Just look that up in the code, it's quite nice commented. </tt> <tt> /* check for completed lines of input */</tt> <tt> ndo2db_check_for_client_input(&idi,&dbuf);</tt> <tt>This function also decides if the actual client needs to be disconnected. So there are 3 main options that can lead to the termination of the child:</tt> <tt>* socket read result fails with -1, eagain/eintr -> <a href="http://linux.die.net/man/2/read">http://linux.die.net/man/2/read</a></tt> <tt>* socket read is empty</tt> <tt>* client input is wrong protocol (idi->disconnect_client)</tt> <tt>If one line has been read from the socket, it comes to ndo2db_handle_client_input where the data frame is analyzed. If clients protocol (ndomod) doesn't match ndo2db protocol, client will be disconnected and data ignored.</tt> <tt> /* client is using wrong protocol version, bail out here... */</tt> <tt> if(idi->protocol_version!=NDO_API_PROTOVERSION){</tt> <tt> syslog(LOG_USER|LOG_INFO,"Error: Client protocol version %d is incompatible with server version %d. Disconnecting client...",idi->protocol_version,NDO_API_PROTOVERSION);</tt> <tt> idi->disconnect_client=NDO_TRUE;</tt> <tt> idi->ignore_client_data=NDO_TRUE;</tt> <tt> return NDO_ERROR;</tt> <tt> }</tt> <tt>Other data will be read, parsed and saved by type as idi->buffered_input ... If the data section is complete given by data_type==NDO_API_ENDDATA then ndo2db_end_input_data will be called wherein the dbhandling starts e.g.</tt> <tt> case NDO2DB_INPUT_DATA_PROGRAMSTATUSDATA:</tt> <tt> result=ndo2db_handle_programstatusdata(idi);</tt> <tt> break;</tt> <tt>values for the dbquery will be read (dbhandler.c:,ndo2db_handle_programstatusdata) and then the queries are sent to db - currently MySQL only, calling ndo2db_db_query</tt> <tt>But the real question is - where is ndo2db_handle_processdata called that will perform the table trimming if program is starting up? Ok, that's the same datahandling.</tt> <tt> /* realtime Nagios data */</tt> <tt> case NDO2DB_INPUT_DATA_PROCESSDATA:</tt> <tt> result=ndo2db_handle_processdata(idi);</tt> <tt> break;</tt> <tt>Looking at the way we got here it's quite simple to see that only the child will call this function. So everytime the child dies and gets forked again, the condition of starting up and outdated realtime data matches. And furthermore, the trimming of the tables. </tt> <tt>-----------------------</tt> <tt>Stepping a bit more on the client side with ndomod, the data comes from </tt> <tt> /* realtime data */</tt> <tt> case NDO_API_PROCESSDATA:</tt> <tt> idi->current_input_data=NDO2DB_INPUT_DATA_PROCESSDATA;</tt> <tt> break;</tt> <tt>which is generated in ndomod.c:ndomod_broker_data</tt> <tt> /* handle the event */</tt> <tt> switch(event_type){</tt> <tt> case NEBCALLBACK_PROCESS_DATA:</tt> <tt>and written to datasink</tt> <tt> /* write data to sink */</tt> <tt> if(write_to_sink==NDO_TRUE)</tt> <tt> ndomod_write_to_sink(dbuf.buf,NDO_TRUE,NDO_TRUE);</tt> <tt>-----------------------</tt> <tt>The conditional match for NEBTYPE_PROCESS_PRELAUNCH comes directly from Nagios and the defines for the Broker module. This type will be sent to the broker after loading all neb modules.</tt> <tt>It seems that your Nagios version does not change this type or it simply reloads the broker module every 60 seconds. It could occur that the nagios process is restarted for whatever reason, since the do while loop runs on sigrestart==TRUE && sigshutdown==FALSE</tt> <tt>But I am not very familiar with that, that's just a guess. The only thing is that the ndomod/ndo2db will receive the NEBTYPE_PROCESS_PRELAUNCH and then do the work. So the conclusion would be that they're working fine. </tt> <tt>The only thing I could imagine at last is that the data written/read from the socket (tcp or unix) is truncated. The defines for the NEBTYPEs only differ by one or two bits. That could lead into errors but every 60 seconds is kind of weird....</tt> <o:p></o:p> <tt> </tt><tt>I am referring to the code in visualization software, not ndo.</tt><o:p></o:p> <tt>Then you should point exactly to it as I am not aware what you meant or which codebase you are using. Furthermore you should mention Nagios and NDOUtils version, operating system and architecture.</tt> <o:p></o:p> <o:p></o:p> <tt> The other strange thing is that ndo2db refork exactly every 60 seconds. Where in the code I can find this interval? Why the child ndo2db exits?</tt><o:p></o:p> <tt>See above, it seems that the event broker/ndomod doesn't send any data (or the socket is broken) and then a refork is performed. </tt> <tt>How did you do that trace?</tt> <tt>Kind regards,</tt> <tt>Michael</tt> <o:p></o:p> </div> </body> </html>