Core 4 Remote Workers

[Andreas Ericsson]

On 02/02/2013 06:24 PM, Daniel Wittenberg wrote:
> Maybe I misunderstood, but it sounds like you are going to have the
> workers listen for work coming from core, rather than the workers
> going and fetching work?

Yes. The scheduler will only be on the master node.

>  If that is the case it would concern me a
> bit as far as being robust, in that if a server goes away, or can't
> talk to a worker anymore, you have to deal with timeouts, and
> potentially cause slow-downs in getting work processed.

Not really, no. The "server goes away" issue is already handled. If
a worker (any worker) crashes its jobs will be distributed to the
other workers in its workergroup.

>  So I always
> liked the idea better that the workers check in and get work from the
> core,

That's exactly what will happen, and what's happening now.

Worker connects to core and says "Hi there! I can do some work!"
Core says "Alrightey!"
Workers says (well, nothing really, but it just listens for work)
Core says "do this!"
worker says "rightey, I'm done".

Core detects immediately if the link goes down though, so this is
actually better than having the workers go out and ask for work,
since that would involve an additional query/response round trip.

> then you remove the need for any listening sockets.

Nopes. Where are the workers supposed to check in to get work if
core isn't listening anywhere? Sure, we can move the listening
socket to a message queue of some sort, but that won't change the
fact that we still have to listen.

>  I think
> you'll also need to consider security in that when a worker does
> check in, how do you know it's not a malicious worker, shared key or
> something, and then of course there's transport security, so ssl or
> payload-level encryption.
> 

libssh2, most likely, with preshared keys the same way you use keys
to do password-less logins via ssh (although using password protected
keys will probably be quite a large pain).

-- 
Andreas Ericsson                   andreas.ericsson at op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_jan