Unix Domain Sockets + Fork for improved scalability

Server Mongrel
1

Hello All,

Ticket #30 (http://mongrel.rubyforge.org/ticket/30) has a set of
changes to mongrel that I’ve written on behalf of
my employer, Raritan Computer, Inc. Also, these changes are released
under the existing Ruby license used for Mongrel.

Raritan is using mongrel internally for product development and was
not able to get the level of scalability desired due to limitations
Mongrel/Rails/Ruby. In particular the single threaded nature of
processing rails request is a serious bottleneck when one or more long
running requests could occur. Originally we used mongrel cluster to
spin up a small number of mongrel rails instance; Apache mod_proxy was
used to hand off requests to the various instances of mongrel.

Unfortunately a long running request could cause our entire web
application to “hang”. As these requests are being processing one or
more mongrel instances continued to accept and enqueue new connections
it can’t immediately service.

Mongrel cluster didn’t provide any features for dynamically sizing the
number of mongrel_rails instances necessary to keep the application
running smoothly. Also, mongrels nature of continuing to enqueue
connections while processing a long running request was a problem for
our application. In essence we needed a smarter mongrel that would
know which instances of a “cluster” were busy or free; if none were
free then it should spin up a new instance.

The attached patch does just that. It’s definitely tailored for a Unix
derivative platform and depends on the functionality of the UNIXSocket
class to pass file descriptors between instances. It works in a
master-slave type of server setup. The “master” is the initial
instance of mongrel rails which goes through its normal initialization
gyrations before forking mulitple “slave” instances to do the actual
rails processing. The server listens on the defined TCP/IP address and
port accepting request (from Apache in our case), determines which
child if any is available, and then hands off the file descriptor for
processing.

The number of children can dynamically grow and shrink depending on
how busy the application is being kept. The patch has fixed our issue
with having long running requests hang the web application.

The patch isn’t without any warts. Due to the way that mongrel
preforms initialization I found it difficult to follow the startup
logic. Because of the convoluted
contortions that occur during startup I was unable to
make deeper changes due to time constraints which means that the
following problem exists.

  • If you start the server as root and the configuration causes a
    setuid/setgid call to a non-root user then during shutdown mongrel may
    not be able to remove its pid file.

I’m not aware of any other lurking problem, but I wouldn’t be
surprised if one or more exists.

If you have any further question just send me a note or post to the
list. I’ll stay subscribed for a few weeks at least. Also, for the
maintainers: I’ve added a copyright notice for Raritan Computer on
those files that had extensive changes. I believe that is the correct
way to handle major changes, let me know if I’m incorrect in my
understanding.

Enjoy

– Brian W.

sigh here is the rest of the text I’m required to include

THIS SOFTWARE IS PROVIDED “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

–

/* insert witty comment here */

2

Hey Brian-

This is pretty kick ass, I’ll definitely take a look.

-Ezra

3

Thanks…

I was just reviewing the patch and I’ve notice several spelling errors
(and grammar problems) in my log messages that need to be fixed. Also,
there isn’t any logic handling any kind of IO error on the TCP Server
socket for the UnixDispatchServer. That could be a problem, but I
don’t really expect any IO errors on the actual TCP socket for
accepting incoming request from Apache (mod_proxy).

I’ll try to review the code changes again after I get some sleep (I
really should have been in bed hours ago) and submit a new patch with
any fixes that I make; cosmetic or otherwise.

– Weave

On Tue, May 6, 2008 at 12:30 AM, Ezra Z. [email protected]
wrote:

Mongrel/Rails/Ruby. In particular the single threaded nature of
Mongrel cluster didn’t provide any features for dynamically sizing the
master-slave type of server setup. The “master” is the initial

Enjoy

Mongrel-users mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/mongrel-users

–

/* insert witty comment here */

4

How does this work with the AR database connections? Are they closed
and reopened in the children? Or do the children load rails after they
fork?

-Ezra

On May 5, 2008, at 10:36 PM, Brian W. wrote:

really should have been in bed hours ago) and submit a new patch with

-Ezra

changes to mongrel that I’ve written on behalf of
was
the
derivative platform and depends on the functionality of the
for
contortions that occur during startup I was unable to

– Brian W.

[email protected]
[email protected]
http://rubyforge.org/mailman/listinfo/mongrel-users

5

On Tue, May 06, 2008 at 01:36:32AM -0400, Brian W. wrote:

Thanks…

I was just reviewing the patch and I’ve notice several spelling errors
(and grammar problems) in my log messages that need to be fixed. Also,
there isn’t any logic handling any kind of IO error on the TCP Server
socket for the UnixDispatchServer. That could be a problem, but I
don’t really expect any IO errors on the actual TCP socket for
accepting incoming request from Apache (mod_proxy).

Well, experience on the project has shown that the second you toss this
thing on FreeBSD you get a whole new cluster of IOError you never knew
existed. Then put it on OSX and you get 10 more. Each all an exception
of wonderful clarity.

I’m pretty sure as people try this out you’ll find them all though.

I’ll try to review the code changes again after I get some sleep (I
really should have been in bed hours ago) and submit a new patch with
any fixes that I make; cosmetic or otherwise.

No! Work now! Sleep later! :slight_smile:

Evil: http://www.zedshaw.com/
Utu : http://savingtheinternetwithhate.com/

6

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian W. wrote:

Hello All,

Ticket #30 (http://mongrel.rubyforge.org/ticket/30) has a set of
changes to mongrel that I’ve written on behalf of
my employer, Raritan Computer, Inc. Also, these changes are released
under the existing Ruby license used for Mongrel.

Interesting Brian, glad you finally got the patch into the queue. :slight_smile:

The attached patch does just that. It’s definitely tailored for a Unix
derivative platform and depends on the functionality of the UNIXSocket
class to pass file descriptors between instances. It works in a
master-slave type of server setup. The “master” is the initial
instance of mongrel rails which goes through its normal initialization
gyrations before forking mulitple “slave” instances to do the actual
rails processing. The server listens on the defined TCP/IP address and
port accepting request (from Apache in our case), determines which
child if any is available, and then hands off the file descriptor for
processing.

So, I’ve just read through this patch, and correct me if I’m wrong, but
your patch removes the ability for Mongrel to run as a plain web server
right? There also seems to be no tests, so does it just use the
existing tests but within the confines of the new forking setup?

The patch isn’t without any warts. Due to the way that mongrel
preforms initialization I found it difficult to follow the startup
logic. Because of the convoluted
contortions that occur during startup I was unable to
make deeper changes due to time constraints which means that the
following problem exists.

Can you describe what was confusing? The code’s not too large so maybe
it just wasn’t documented really clearly.

Also, had you tried just subclassing HttpServer and implementing the
alternative mechanism for both the master and slaves?

  • If you start the server as root and the configuration causes a
    setuid/setgid call to a non-root user then during shutdown mongrel may
    not be able to remove its pid file.

The pid file management has always been a mess because of ruby’s very
poor process management, and just the variablility in POSIX anyway.
Eventually it’ll just have to be a C api that does it right I think.

I’m not aware of any other lurking problem, but I wouldn’t be
surprised if one or more exists.

If you have any further question just send me a note or post to the
list. I’ll stay subscribed for a few weeks at least. Also, for the
maintainers: I’ve added a copyright notice for Raritan Computer on
those files that had extensive changes. I believe that is the correct
way to handle major changes, let me know if I’m incorrect in my
understanding.

Actually, and this is a big misnomer among open source folks, the
copyright isn’t on a file, it’s on the project. It’s kind of like
saying if you edit a page in my book with a marker you get to put your
copyright on that page. Additionally there isn’t enough change to that
file to warrant an entire copyright change as it’s still primarily
written by myself and other project contributors.

What you can do is either officially donate it back to the project under
the original copyright with a small notice giving credit to yourself
and/or Raritan that you did the master/slave/unixsocket hack, or pull
this out into a separate, completely and wholely written by you/raritan
file that has none of my code and then release under your own license
however you like.

Talk it over with your corporate masters, but considering you have an
entire product based on other people’s generously donated free work it
might be a really great nice thing to give back.

Makes people all warm and fuzzy and like your products in exchange.

Evil: http://www.zedshaw.com/
Utu : http://savingtheinternetwithhate.com/

7

See the comments embedded…

On Tue, May 6, 2008 at 1:59 AM, Zed A. Shaw [email protected] wrote:

So, I’ve just read through this patch, and correct me if I’m wrong, but
your patch removes the ability for Mongrel to run as a plain web server
right? There also seems to be no tests, so does it just use the
existing tests but within the confines of the new forking setup?

No, the patch should not remove the ability for mongel to run a a
plain web server. I basically split the HttpServer class into two
classes: Server and HttpServer. I attempted to move most of what would
be common code between the existing HttpServer class and the new
UnixDispatchServer class into the base Server class. If you specify
‘–unixmaster’ or put ‘unixmaster = true’ in the mongrel yaml
configuration file then it runs an instance of the UnixDispatchServer;
otherwise it should run the default HttpServer as it was originally
written to do,

Your right there are currently no test. Yes you can flame me all you
want for that; I’ll get to it when I have time. I figured in the
spirit of most open source projects I would go ahead and “release
early” to let people play with it so that I could get feedback. The
primary development platform was Linux (CentOS 4 actually) and I
tested it heavily with our application before I posted it. It’s far
from perfect, but it is a start.

Besides other people might feel like contributing, I wouldn’t want to
deprive them of a decent opportunity by writing all the test myself.
That would seem kind of greedy. :wink:

it just wasn’t documented really clearly.
poor process management, and just the variablility in POSIX anyway.
Eventually it’ll just have to be a C api that does it right I think.

Almost all systems support the ‘unlink’ command. At issues is when you
drop privilege levels from root after creating a pid file then it’s
almost impossible to remove. I’ve gotten around this issue by using a
service script (/etc/init.d/mongrel_rails) on our system to check for
the existence of a pid file, ensure that the process really isn’t
running, and then removing the pid before starting up mongrel. It’s
not an insurmountable problem, I just figured I would state it up
front to save other developers and users some time.

One could always change the code to save the privilege level allowing
for a new call to setuid/setgid, but this could come back to bite if
the process has a vulnerability. It’s a small problem to do the checks
before starting or after stopping mongrel for the added security of
running as a non-privileged user.

Actually, and this is a big misnomer among open source folks, the
copyright isn’t on a file, it’s on the project. It’s kind of like
saying if you edit a page in my book with a marker you get to put your
copyright on that page. Additionally there isn’t enough change to that
file to warrant an entire copyright change as it’s still primarily
written by myself and other project contributors.

I’m not a copyright lawyer, but I bet if you talk to a few you would
get some variance on how it really should be done. The point is the
work I’ve done has been for Raritan. If the changes are such that
copyright law would assign the ownership, for all or part of the
changes, to Raritan then that’s who owns it. Raritan is simply trying
to follow the Ruby license that Mongrel is using…

Specifically section 2a of the license at
http://www.ruby-lang.org/en/LICENSE.txt

might be a really great nice thing to give back.

Makes people all warm and fuzzy and like your products in exchange.

I know all about warm and fuzzy, giving to the community, paying for
the privilege, and being out of work with no income at the end of the
day. Luckily the work lives own and my previous company was able to
find someone to keep the open source fork going while we tried our
commercial endeavor. The commercial endeavor was a bust, but that’s
life. If you have any interest in network management then you should
hop on over to OpenNMS (http://www.opennms.org/) and you can see my
first dabbling with open source.

– Weave

–

/* insert witty comment here */

8

Mongrel cluster didn’t provide any features for dynamically sizing the
number of mongrel_rails instances necessary to keep the application
running smoothly. Also, mongrels nature of continuing to enqueue
connections while processing a long running request was a problem for
our application. In essence we needed a smarter mongrel that would
know which instances of a “cluster” were busy or free; if none were
free then it should spin up a new instance.
Reminds me of swiftiply vaguely.

9

On Tue, May 6, 2008 at 1:59 AM, Zed A. Shaw [email protected] wrote:

this out into a separate, completely and wholely written by you/raritan
file that has none of my code and then release under your own license
however you like.

We haven’t required copyright assignment from any of the other
contributors, so there’s no way we could relicense anyway without
stripping out their code. So I think it is fine for Raritan to keep
the original copyright, since it’s under the same license.

I agree that there’s no point in the additional copyright notice,
though; Raritan is just one of the many (some anonymous now)
contributors.

10

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian W. wrote:

Unfortunately a long running request could cause our entire web
application to “hang”. As these requests are being processing one or
more mongrel instances continued to accept and enqueue new connections
it can’t immediately service.

Another solution:

You could stick HAproxy in front of your mongrel cluster with a the
configuration somewhat like:

listen application *:80
balance roundrobin
server mongrel0 127.0.0.1:5000 minconn 1 maxconn 1 check
server mongrel1 127.0.0.1:5001 minconn 1 maxconn 1 check
server mongrel2 127.0.0.1:5002 minconn 1 maxconn 1 check
server mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
server mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check

The ‘minconn 1 maxconn 1’ will force haproxy to queue the results within
itself instead of mongrel, and the ‘check’ will take the mongrel out of
rotation if it goes down, and add it back into the rotation as soon as
it comes back up. As soon as one mongrel is finished with a request,
haproxy will give it one of the ones it has queued.

This is a simple version of what I’m doing to load balance clusters of
mongrels these days.

enjoy,

-jeremy

–

Jeremy H. [email protected]

11

On Thu, May 08, 2008 at 12:50:25AM -0700, Eric W. wrote:

monitoring/testing something against an individual backend Mongrel
process far more difficult as I require Mongrel to accept connections
haproxy didn’t forward; so I disabled concurrency on the Mongrel
side instead.

And we can’t forget about ngx_http_upstream_fair_module.c either.

Thanks Grzegorz and Ezra and anyone else who worked on that.

enjoy,

-jeremy

–

Jeremy H. [email protected]

12

Jeremy H. [email protected] wrote:

On Mon, May 05, 2008 at 11:37:56PM -0400, Brian W. wrote:

Unfortunately a long running request could cause our entire web
application to “hang”. As these requests are being processing one or
more mongrel instances continued to accept and enqueue new connections
it can’t immediately service.

Heh, this is also the exact problem I designed qrp around, too.

  server  mongrel3 127.0.0.1:5003 minconn 1 maxconn 1 check
  server  mongrel4 127.0.0.1:5004 minconn 1 maxconn 1 check

The ‘minconn 1 maxconn 1’ will force haproxy to queue the results within
itself instead of mongrel, and the ‘check’ will take the mongrel out of
rotation if it goes down, and add it back into the rotation as soon as
it comes back up. As soon as one mongrel is finished with a request,
haproxy will give it one of the ones it has queued.

This is a simple version of what I’m doing to load balance clusters of
mongrels these days.

I initially tried doing this with haproxy before I made qrp, but it made
monitoring/testing something against an individual backend Mongrel
process far more difficult as I require Mongrel to accept connections
haproxy didn’t forward; so I disabled concurrency on the Mongrel
side instead.

13

Brian,

I found your patch while looking for a solution that we’ve needed to
decrease startup times, take advantage of COW, have completely graceful
restarts ( < 0.1 seconds downtime max, new server must be ready to
immediately start processing requests before the old server is booted
out ), and scheduling of garbage collection between results.

Your patch turned out to be a very good fit for all 4 of these arenas :slight_smile:

Though, I did need to do some refactoring and additional work on what
you’ve started.

The main parts of my improvements are the “-R” flag
(Mongrel::UnixDispatchServer only), graceful shutdowns, and ability to
specify pre/post fork hooks. The -R flag is like a hostile takeover
restart. A new server boots up, gets all loaded and ready to go, then
sends a TERM signal to the old process. The old process IMMEDIATELY
stops listening on it’s port, and waits for it’s children to finish
their requests. As soon as they are all done, the server shuts down.

My changes are available here in my fork of mongrel:

Here’s the script I use to start up mongrel:

http://pastie.org/245416

I’d still like to see some additional cleanup and get some input on the
changes I’ve made. Additionally, I did all this while working for my
company Lead Media Partners, so they should get a plug for this
contribution.

Peace and harmony,

Tim

14

This looks interesting.

A question: can this version be made to run in the foreground, like
`httpd
-DFOREGROUND’? I’d like to use daemontools to control the master
process,
which requires the master to not fork and background itself but stay in
the
foregound and respond to signals.

15

Jos B. wrote:

This looks interesting.

A question: can this version be made to run in the foreground, like
`httpd
-DFOREGROUND’? I’d like to use daemontools to control the master
process,
which requires the master to not fork and background itself but stay in
the
foregound and respond to signals.

It’s run fine for me either in the foreground or daemonized.

16

On Fri, Aug 01, 2008 at 08:28:18AM +0200, Tim H. wrote:

It’s run fine for me either in the foreground or daemonized.

Great, thanks Tim.