ndoutils with zlib

[Andreas Ericsson]

Tilo Renz wrote:
> Hello Developers,
> 
> we are happily using ndomod on more than 1000 machines distributed to
> multiple locations, but the traffic (over leased lines) is
> tremendous. To reduce this traffic I was told to embed zlib into
> ndoutils (ndoutils-1.4b7).
> 

Umm. Are you *monitoring* 1000 machines or are you using 1000 machines to do
the monitoring? Just curious.. :)

Besides that, why not use lzo instead? It's a lot faster (although it doesn't
compress as well), and it comes with a minilzo.c that can easily be included
into the sources for those platforms where the zlib header files aren't
installed.

> What did I do?
> 
> To make ndomod.c more readable, I refactored some of its
> functionality.
> 
> First, I moved the option-parsing to extra files parseopt.[c|h] 
> parseopt.h defines a struct to hold all configuration data for
> ndomod, replacing various global variables in ndomod.c. parseopt.c
> does the option-parsing with parameters given by nagios and the
> configuration file. It also contains the default values, formerly
> kept in ndomod.c.

This sounds like a worthy change in its own right, imo.
It should probably have been a patch of its own. I'll see what I
can make of it.

> For zipped tcp Streams, the option output_type now understands the
> new value 'tcpsocketzip'.
> 
> Second, I refactored io.[c|h]. Handing filedescriptors around is no
> option, when you need a gz_file-Structure. In order to fix this I
> declared a new struct ndo_sink_description_struct, which is holding
> the filedescriptor, the pointer to gz_file and some status-variables,
> intended to replace sink-related global variables in ndomod.c
> (is_open, previously_open). Also io.h defines the new sink-type
> NDO_SINK_TCPSOCKETZIP.
> 
> I replaced all sink-related functions (ndo_sink_open, ndo_sink_write,
> ndo_sink_write_newline, ndo_sink_flush, ndo_sink_close) by new
> functions (ndo_sink_init2, ndo_sink_open2, ndo_sink_write2,
> ndo_sink_write_newline2, ndo_sink_flush2, ndo_sink_close2) which take
> a pointer to a ndo_sink_description_struct as their first argument. I
> admit, this is kind of object-oriented, but not knowing all compilers
> and target platforms of nagios I resisted zo use real Objects.

Umm... C doesn't have what php/java/c++/C# call "real objects", but
OO programming is still quite possible, although the objects are all
created at runtime by assigning function pointers to structs.

The linux kernel makes heavy use of OO programming for the device driver
API.


> I also
> tried to keep the strange indention scheme, although editor & brain
> support seems broken, as the mixed usage of tabs and spaces in the
> given sources prooves. (In real life I'm a wimpy java programmer, so
> I prefer KR)
> 

Heh :P

> Compression works best, if there is a some amount of data to
> compress, so I introcduced some slight changes to ndomod.c concerning
> the (re-)opening of the sink: If the sink-Buffer is empty (and if the
> sink prefers blockwise data, as zipped tcp does) the connection will
> be closed. It will be reopened after the usual reconnect interval.
> (src/ndomod.c line 596ff) This implies that reconnect_time and
> output_buffer_items should be set to reasonable values, otherwise
> data will get lost. Also the logfile may contain some unnecessary
> warnings.
> 
> I patched all other programs (ndo2db, file2sock, log2ndo) according
> to the new possiblities. ndo2db will automatically detect a zipped
> tcp stream, file2sock and log2ndo both have changed command line
> parameters. I also altered their help-texts.
> 
> Finally I _tried_ to fix the build-process. (Here I need help) I
> managed to extend src/Makefile.in and include/config.h.in. 
> config.h.in now has a section for zlib, similar to those for the
> databases. I peppered the whole project with #ifdef HAVE_ZLIBs, so
> that it will compile (and work!) with and without HAVE_ZLIB.
> 

I'll say it again: minilzo.c would have been a ton better for you :-/

> This is where I need your help: I was unable to adapt the
> configure-Script, so that it automatically recognizes if zlib-support
> could be built in. Currently, after ./configure, I have to fix the
> created include/config.h to #define USE_ZLIB 1 (line  272 ff)
> 
> Please help me to fix this issue.
> 
> I would be happy if these changes would be included into a new
> version of ndoutils. Also I am open to your comments and further
> improvements. Beyond the configure-issue the code is working in our
> current setup for several days.
> 

Since this is intended to be a performance booster, I'd like to see
some real numbers on what the performance was before and how this
patch changes it. If it's major code-surgery for a 10% increase, I
think it's safe to say it'll be dropped on the floor, but if it ups
performance by several orders of magnitude, I'd be very interested
to work on it. Either way, some real numbers, please.

> I tried to build a .diff-file, but I doubt the attached file is
> really usefull. I published the whole changed ndoutils at
> http://joey.in-ulm.de/2008/ndoutils Please have a look at it.
> 

It looks ok-ish, although far too large. I wish you'd have made
several different patches of it so that each independant step
could be reviewed and tested for correctness. As it stands now,
it's two days worth of work to just review the entire patch,
simply because the scope of the changes is so huge.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/