A fix for EXTREME slowness

[Cary Petterborg]

I'm resending this email because there was not a single response to my
previous email. I have to think that someone else has run into this
problem, and I would like to know what others have done and suggestions
for the implementation of a good fix. We have solved this problem in the
short term, but we want to implement a more robust long terms solution.
We had a huge performance increase from fixing the problem, so if you
have noticed your web server taking a long time to process your
status.cgi and extinfo.cgi page requests, please read on.....


This email has a description of the problem, the symptoms, our interim
fix, and a possible long term fix. If you have been noticing large (or
larger) load times for status.cgi and/or extinfo.cgi, please read this
entire message.


We have recently had our comments.dat file grow to a much larger size
(due to increased need for comments). This file grew to about 4.8MB. To
read or write this size of file is not a problem, but the processing of
it in status.cgi and extinfo.cgi was slowing things down significantly.
To give you an idea, the page load times went from a few seconds to over
a minute on our production systems.

Since the load times were so bad we started looking for the cause. It
became evident that it was the processing of the comments.dat file. We
created a program to take the comments more than 30 days old and archive
them into an archive file. The reduces the load time so significantly
that we decided to do some tests on a non-production system.

We took the large 4.8MB file and reduced the number of entries until
there were only 30 days worth in the file (down to 90, 80, 70, 60, 50,
40 and finally 30 days). Then we ran tests on status.cgi for each of
these filesizes. Using just a crude stopwatch we measured the times it
took to load the various pages. I have created a spreadsheet file and
graph for the data. The test seems to indicate that the size of the
comments.dat file dramatically affects the page load times. On the test
server, the load times for the 4.8MB file were in the 9 to 10 second
range, while the 2MB file were under 2 seconds. Here is a table of the
results:

File size | Time
-----------------
4.85      | 9.5
3.95      | 6.0
3.00      | 2.8
2.03      | 2.0

This seems to show rather exponential growth rather than linear. We have
ended up in the short term archiving the old data, reducing the file to
the much more reasonable 2MB size and cutting the times significantly.

The results on the production server is even more dramatic reducing the
load time from 70 seconds to about 3 seconds. This was more of a problem
on the production system because there were more status.cgi processes
running at the same time. A 95% reduction in the load time is very
significant.

Are there others who have seen this as a big problem, or is it not a
typical problem that has been encountered? Have others found a way to
fix this problem other than reducing the number of comments in the
comments file?

So there seems to be a need to make this information be more a database
type access, rather than a "parse this big file and see what drops out
that we want" access. This could easily be done with a real relational
database, or even a more simple database, to retrieve only the comments
for the host/service desired. We are willing to do the work on this, but
would like it to be incorporated into Nagios code base so that we are
not having to port this functionality on upgrades in the future.

If you are interested in this type of enhancement, please let me know.
In addition, if you have suggestions for the implementation of real
comments database (yes, we are experienced in this area, and have OUR
ideas of how we want to implement it, but we'd like to know of other
opinions so that we can increase the likelihood of it being incorporated
into the standard release), please let me know.

Thanks!

Cary Petterborg


----------------------------------------------------------------------
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Nagios-users mailing list
Nagios-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nagios-users
::: Please include Nagios version, plugin version (-v) and OS when reporting any issue. 
::: Messages without supporting info will risk being sent to /dev/null