Slave-1.1.0

SYNOPSIS

the Slave class forks a process and starts a drb server in the child
using
any object as the server. the process is detached so it is not
required
(nor possible) to wait on the child pid. a Heartbeat is set up
between the
parent and child processes so that the child will exit of the parent
exits
for any reason - preventing orphaned slaves from running
indefinitely. the
purpose of Slaves is to be able to easily set up a collection of
objects
communicating via drb protocols instead of having to use IPC.

typical usage:

 slave = Slave::new{ AnyObject.new }

 slave.object                  #=> handle on drb object
 slave.uri                     #=> uri of the drb object
 slave.socket                  #=> unix domain socket path for drb 

object
slave.psname #=> title shown in ps/top

 object = slave.object

 value = object.any_method     #=> use the object normally

slaves may be configured via the environment, the Slave class, or via
the
ctor for object itself. attributes which may be configured include

 * object : specify the slave object.  otherwise block value is 

used.
* socket_creation_attempts : specify how many attempts to create a
unix domain socket will be made
* debug : turn on some logging to STDERR
* psname : specify the name that will appear in ‘top’ ($0)
* at_exit : specify a lambda to be called in the parent when the
child dies
* dumped : specify that the slave object should not be
DRbUndumped (default is DRbUndumped)
* threadsafe : wrap the slave object with ThreadSafe to implement
gross thread safety

URIS

http://rubyforge.org/projects/codeforpeople/
http://codeforpeople.com/lib/ruby/slave

HISTORY

1.1.0:
- replaced HeartBeat class with LifeLine.

 - __HUGE__ cleanup of file descriptor/fork management with tons of 

help
from skaar and ezra. thanks guys!

 - introduced Slave.object method used to return any object 

directory from
a child process. see samples/g.rb.

 - indroduced keyword to automatically make slave objects 

threadsafe.
remember that your slave object must be threadsafe because they
are
being served via DRb!!!

SAMPLES

<========< samples/a.rb >========>

~ > cat samples/a.rb

 require 'slave'
 #
 # simple usage is simply to stand up a server object as a slave. 

you do not
# need to wait for the server, join it, etc. it will die when the
parent
# process dies - even under ‘kill -9’ conditions
#
class Server
def add_two n
n + 2
end
end

   slave = Slave.new :object => Server.new

   server = slave.object
   p server.add_two(40) #=> 42

   slave.shutdown

~ > ruby samples/a.rb

 42

<========< samples/b.rb >========>

~ > cat samples/b.rb

 require 'slave'
 #
 # if certain operations need to take place in the child only a 

block can be
# used
#
class Server
def connect_to_db
“we only want to do this in the child process!”
@connection = :postgresql
end
attr :connection
end

   slave = Slave.new('object' => Server.new){|s| s.connect_to_db}

   server = slave.object

   p server.connection  #=> :postgresql
 #
 # errors in the child are detected and raised in the parent
 #
   slave = Slave.new('object' => Server.new){|s| s.typo} #=> raises 

an error!

~ > ruby samples/b.rb

 :postgresql
 ./lib/slave.rb:460:in `initialize': undefined method `typo' for 

#Server:0xb7565694 (NoMethodError)
from samples/b.rb:22:in `new’
from samples/b.rb:22

<========< samples/c.rb >========>

~ > cat samples/c.rb

 require 'slave'
 #
 # if no slave object is given the block itself is used to contruct 

it
#
class Server
def initialize
“this is run only in the child”
@pid = Process.pid
end
attr ‘pid’
end

   slave = Slave.new{ Server.new }
   server = slave.object

   p Process.pid
   p server.pid # not going to be the same as parents!
 #
 # errors are still detected though
 #
   slave = Slave.new{ fubar } # raises error in parent

~ > ruby samples/c.rb

 14387
 14388
 ./lib/slave.rb:460:in `initialize': undefined local variable or 

method fubar' for main:Object (NameError) from samples/c.rb:21:in new’
from samples/c.rb:21

<========< samples/d.rb >========>

~ > cat samples/d.rb

 require 'slave'
 #
 # at_exit hanlders are handled correctly in both child and parent
 #
   at_exit{ p 'parent' }
   slave = Slave.new{ at_exit{ p 'child' };  'the server is this 

string’ }
#
# this will print ‘child’, then ‘parent’
#

~ > ruby samples/d.rb

 "child"
 "parent"

<========< samples/e.rb >========>

~ > cat samples/e.rb

 require 'slave'
 #
 # slaves never outlive their parent.  if the parent exits, even 

under kill -9,
# the child will die.
#
slave = Slave.new{ at_exit{ p ‘child’ }; ‘the server is this
string’ }

   Process.kill brutal=9, the_parent_pid=Process.pid
 #
 # even though parent dies a nasty death the child will still print 

‘child’
#

~ > ruby samples/e.rb

 "child"

<========< samples/f.rb >========>

~ > cat samples/f.rb

 require 'slave'
 #
 # slaves created previously are visible to newly created slaves - 

in this
# example the child process of slave_a communicates directly with
the child
# process of slave_a
#
slave_a = Slave.new{ Array.new }
slave_b = Slave.new{ slave_a.object }

   a, b = slave_b.object, slave_a.object

   b << 42
   puts a #=> 42

~ > ruby samples/f.rb

 42

<========< samples/g.rb >========>

~ > cat samples/g.rb

 require 'slave'
 #
 # Slave.object can used when you want to construct an object in 

another
# process. in otherwords you want to fork a process and retrieve a
single
# returned object from that process as opposed to setting up a
server.
#
this = Process.pid
that = Slave.object{ Process.pid }

   p 'this' => this, 'that' => that

 #
 # any object can be returned and it can be returned asychronously 

via a thread
#
thread = Slave.object(:async => true){ sleep 2 and [ Process.pid,
Time.now ] }
this = [ Process.pid, Time.now ]
that = thread.value

   p 'this' => this, 'that' => that

~ > ruby samples/g.rb

 {"that"=>14406, "this"=>14405}
 {"that"=>[14407, Tue Nov 28 09:47:31 MST 2006], "this"=>[14405, Tue 

Nov 28 09:47:29 MST 2006]}

enjoy.

-a

On 11/28/06, Ara.T.Howard [email protected] wrote:

communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

TwP

On Nov 28, 2006, at 1:49 PM, Tim P. wrote:

between the
parent and child processes so that the child will exit of the
parent exits
for any reason - preventing orphaned slaves from running
indefinitely. the
purpose of Slaves is to be able to easily set up a collection
of objects
communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

Slave is UNIX only since it relies on fork(). I’ve looked at the
code. Ara does a nice job of keeping the fork stuff encapsulated. If
you have a Win32 mechanism that could be bolted on to Slave and
replace fork(), you’d get the rest of the code for free.

cr

On Wed, 29 Nov 2006, Tim P. wrote:

parent and child processes so that the child will exit of the parent
exits
for any reason - preventing orphaned slaves from running indefinitely.
the
purpose of Slaves is to be able to easily set up a collection of objects
communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

in all seriousness i’d be happy if it could be made to work - have a
look and
let me know what you think if you get a chance.

-a

On 11/28/06, [email protected] [email protected] wrote:

On Wed, 29 Nov 2006, Tim P. wrote:

Does Slave work on the Win32 platform, or is this a UNIX only gem?

in all seriousness i’d be happy if it could be made to work - have a look and
let me know what you think if you get a chance.

I think Daniel B. is your man for getting this to work on Windows.
Just one of those things that would be “nice to have” in the future.

TwP

On Wed, 29 Nov 2006, Tim P. wrote:

parent and child processes so that the child will exit of the parent
exits
for any reason - preventing orphaned slaves from running indefinitely.
the
purpose of Slaves is to be able to easily set up a collection of objects
communicating via drb protocols instead of having to use IPC.

Does Slave work on the Win32 platform, or is this a UNIX only gem?

what is this “windows” you speak of?

:wink:

-a

Tim P. wrote:

Just one of those things that would be “nice to have” in the future.
It’s probably doable, but I’ll have to look at the source code.

I don’t know much about slave, but one question I have right off the
bat is why you would want one drb server per object, instead of just
one drb server that stored objects in a hash table that you could push
and pop at will. Maybe I need to go back and see Ara’s use case for
this…

Regards,

Dan

Tim P. wrote:

TwP

Yes, but why? I assume the purpose is IPC. If so, again I ask why you
would need 1 drb server per object instead of one drb server with lots
of objects. If its purspose isn’t IPC, what is it?

Thanks,

Dan

On 11/28/06, Daniel B. [email protected] wrote:

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

On Wed, 29 Nov 2006, Daniel B. wrote:

Slave forks off a ruby object in a separate process and then uses DRb
for communication between the processes. Each forked process has its
own DRb server.

TwP

Yes, but why? I assume the purpose is IPC. If so, again I ask why you
would need 1 drb server per object instead of one drb server with lots
of objects. If its purspose isn’t IPC, what is it?

the purpose is two-fold:

  1. easy ipc
  2. scalling expensive operations across multiple processors easily

you certainly could setup a hashtable of objects. but a few things to
consider

  1. drop-in scalibility
 object = ExpensiveCalcuation.new

 # 1000 lines of code

change to

 object = Slave.new{ ExpensiveCalcuation.new }

 # exact same 1000 lines of code

and, whammo. you scale to multi-processor machines.

  1. DRbUndumped nuances
 let's say you build a table of objects like this:


   harp:~ > cat a.rb
   require "slave"
   require "yaml"

   slave = Slave.new{ Hash.new }
   table = slave.object

   array = []

   table["array"] = array

   table["array"] << Process.pid

   y "slave.pid" => slave.pid
   y "array" => array
   y "table['array']" => table["array"]


 what will happen?


   harp:~ > ruby a.rb
   slave.pid: 5258
   array: []
   table['array']: []


 yikes!!!  what went wrong?  well, consider:

   array = []                          # array is in parent

   table["array"] = array              # array is marshaled across 

to child as a copy

   table["array"] << Process.pid       # array is marshaled back 

across to parent and this local_copy is appended too

   p "array" => array                  # local copy empty

   p "table['hash']" => table["hash"]  # remote copy empty


 here is a fix

   array = []
   array.extend DRbUndumped            # the important part


 now, running again


   harp:~ > ruby a.rb
   slave.pid: 5302

   array:
   - 5301

   table['array']:
   - 5301


 but look carefully  this example is even __more__ fubar: the array 

lives in
the parent! we setup a table and sent only the proxy (DRbUndumped)
across
to the child! we’ve completely negated the entire benefit of
running
objects in another process - all we have is a lookup table for
objects in
our own address space.

 ok.  take two.  this time we'll try this:

   harp:~ > cat a.rb
   require "slave"
   require "yaml"

   slave = Slave.new{     # build table in child
     table = Hash.new
     array = []
     table["array"] = array
     table
   }

   table = slave.object

   table["array"] << Process.pid

   y "slave.pid" => slave.pid
   y "table['array']" => table["array"]


 we give a whirl:


   harp:~ > ruby a.rb
   slave.pid: 5358
   table['array']: []


 sigh.  now our table exists only in the child, and so does all it's
 elements.  why doesn't it work.  DRbUndumped is screwing us again. 

here’s
why:

   table["array"] << Process.pid         # here the child marshals a 

copy back to
# parent. we append to the
copy

   y "table['array']" => table["array"]  # and here we get a fresh 

empty copy

 dang.  we know the fix though, in the child we do:

   array.extend DRbUndumped


 trying again

   harp:~ > ruby a.rb
   slave.pid: 5405
   table['array']: !ruby/object:DRb::DRbObject
     ref: -609755446
     uri: drbunix:///tmp/hash_-609755436_5404_5405_0

 wtf?  well, since our object now lives in the child and we only get 

a drb
proxy back some things work, like ‘<<’, but other methods, like
‘to_yaml’, and
‘inspect’ are handled locally by drb. to get around this ‘to_yaml’
bug we have
to force a copy

   harp:~ > cat a.rb
   require "slave"
   require "yaml"

   slave = Slave.new{     # build table in child
     table = Hash.new
     array = []
     array.extend DRbUndumped
     table["array"] = array
     table
   }

   table = slave.object

   table["array"] << Process.pid

   y "slave.pid" => slave.pid
   y "table['array']" => table["array"].map          # force local 

copy, neither dup nor clone will cut it!

   harp:~ > ruby a.rb
   slave.pid: 5465
   table['array']:
   - 5464


 ah, finally.

so, you see, managing a hierarchy of object across drb is tricky. not
that it
can’t be done - i just happen to think that managing one object and
it’s
DRbUndumped nuances is about all most hackers can manage - thus that’s
the
default usage. nothing will stop you from making your server object a
factory
or table though - it just gets confusing fast.

kind regards.

-a