More Threads Please?

[Andreas Ericsson]

Steven D. Morrey wrote:
> Hello Everyone,
> 
> I've been profiling Nagios and found that at least on my installation
> with 30,000+ checks  that 60% of the time is spent in reaping events
> and 3% is spent in saving status and another 1%  is spent in writing
> out retention files. This means that less than 30% is actually spent
> executing checks, to compound the problem the latency is really,
> really high and it looks like it's due to high priority events
> blocking execution. It seems to me that all of these "high priority"
> events are high priority because they need to run right now, but I
> don't see why they need to block execution of everything else just to
> perform housekeeping tasks.
> 
> Assuming I were to put a mutex on the relevant objects, can anyone
> foresee a problem with running these high priority events in their
> own thread or even threads?
> 

Not off-hand, no.


> I have 2 designs in mind. One would be a simple refactor that calls
> handle_timed_event with pthread create, on the high priority events. 
> This would create a new thread for each high priority event, but
> would be literally be a change to only a couple lines of code.
> 

That would be massively silly though, since it would then spend quite
a lot of time creating threads instead of making checks.

> Another design would call for a breaking the main event loop into a
> high priority thread and a low priority thread. In this case we would
> only have 2 threads, we loose a bit of the flexibility (and danger)
> and of having per event threads for high priority events, we also
> loose most of  the overhead of thread creation.
> 

I like this the best. The two threads could just spin (gently) on their
queues until there's something for them to do. Don't implement the
spinlock with mutexes though, as they're sometimes implemented as
"hard" spinlocks.

> I'm not sure high priority events are frequent enough that thread
> creation overhead is significant.
> 

They can be when Nagios is very busy, which is where we least want
the overhead of creating them.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july