[PATCH] Re: alternative scheduler

[Andreas Ericsson]

On 12/03/2010 12:20 PM, Fredrik Thulin wrote:
> On Fri, 2010-12-03 at 11:40 +0100, Andreas Ericsson wrote:
>> ...
>>> What is sched_yield? I can't find that function anywhere in the source
>>> code. Feel free to improve the patch - as I've previously said C isn't
>>> my game.
>>>
>>
>> sched_yield() causes the kernel to check through its scheduling queue and
>> see if there are other processes waiting to run. If there are, those other
>> processes will run. If not, the current process will continue running.
> 
> As I see it, the Nagios scheduler can't afford to miss the opportunity
> to start another check,

Sensible thinking, sort of. I think Max Schubert posted a patch to make it
work like you want if large installation tweaks are enabled. That seems
pretty sound, since large installations generally have dedicated systems
for monitoring purposes whereas smaller installations commonly share hw
with other servers.

> but I'm not going to protest if you prefer
> 
> if (...) {
>    shed_yield();
>    continue;
> }
> 

I'll look into it and make some suitable change that prevents hogging the
cpu entirely but still removing the (way too long) sleep time.

>>> My first scheduler ticked once per second and *BAM* started 30+ checks.
>>>
>>> A lot of the times, a significant number of these checks were exactly
>>> the same check (but different target hosts), so my theory is they all
>>> requested the very same resources around the same millisecond. When I
>>> changed the scheduler to start one check every 50 ms instead, I saw that
>>> I could start around 25% more checks every second. Other theories are
>>> welcome, but that was my observation.
>>>
>>
>> The problem is the tick-time. I'm guessing you fired the checks and then
>> did sleep(1) (or whatever the erlang equivalent is), but that means you
>> lose a couple of milliseconds each second (the time it takes to fire up
>> the checks), which will inevitably cause you to drift in the scheduler.
>> All such sleep()-alike calls are implemented in the kernel with a TICK
>> precision that varies from system to system. Most systems have a 10 usec
>> tick-rate, so if you start sleeping at 1.94 seconds and sleep for one
>> second you'll end up at 2.94 instead of, as a scheduler would wish, at
>> 2.0 when checks are actually scheduled.
> 
> No, actually not. Erlang is a soft real time system. My approach was to
> ask the Erlang VM to send me a tick every N ms (N = 300s * 1000 / number
> of checks). So if N is 50, the VM will signal me once every 50 ms, very
> precisely and without any drift.
> 

If N is constant, it can't be the lvalue of the above expression.

> I then just had to finish starting another check command in =<  49 ms,
> and go back to sleep. All handling of check results is done completely
> asynchronous to this starting of new checks.
> 
> This is all in src/npers_spawner.erl if anyone is interested in the
> details.
>

That's still "doing more than you did before", on a system level, so the
previous implementation must have been buggy somehow. Perhaps erlang
blocked a few signals when the signal handler was already running, or
perhaps you didn't start enough checks per tick?

If the above expression was correct (N is not constant), this algorithm
makes the cost for running a single check exponential with the number of
checks to run. Ie, the more checks you have, the more expensive each check
will become. The curve will converge on (infinity - 1) faster with a larger
exponent. In this case, the exponent is ticks/sec, so reducing the ticktime
means you're effectively reducing performance unless there are other
factors involved that shaves enough cycles to make this change disappear
in the noise.

> ...
>> I'll see about adding something similar to your patch to the scheduler.
>> It's a good one in spirit, but the implementation left a little to be
>> desired.
> 
> Thanks! It would really make my life easier if the patch was in the next
> Nagios release Ubuntu ships =).
> 

That I can't guarantee. It'll be in the next op5 release though, due out
in 12 days.

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev