Problems with Mongrel + Pound

Hi,

I have been using mongrel + Pound for the past 6 months on a single
server deployment.

Recently mongrel has started to behave very weirdly after we?moved app
to a bunch of servers from a single server deployment.

  • It crashes on every exception, throws 500 and throws 500 for ALL
    instances of the cluster? after one exception has occurred
  • Connectivity to SQL seems to take a rather long time (the db is a
    different server on the same network, servers are connected through a
    gigaport switch)
  • Images deliveries have slowed down since we migrated from single
    server to multi server deployment (still using pound, not yet moved to
    apache / nginx)
  • The application architecture uses lot of background processes?- built
    using?DRB which again are deployed physically on different dedicated
    machines to distribute load
  • ?Mongrel doesn’t always log the error 500 within its logs (mongrel may
    not be getting these requests).
    -? Other mongrel instances in the same cluster do not respond even if
    one mongrel has crashed or has encountered an exception.

Here is some basic info of the deployment

3 web servers - Dualcore Dell servers with Xeon 2.0 ghz processor
running Cent OS 5
4 GB Ram Each
1 seperate components server - for threaded Drbs, same config as above
1 seperate database server (not yet clustered, same config as above)

I would appreciate if anyone can analyze the issues and provide me a
solution as it is becoming frustrating to have too much down time in the
application

regards

Mohammed

What version of ruby are you running

ruby -v

Dave,

Here is the version of Ruby along with other Gems installed

ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]

*** LOCAL GEMS ***

abstract (1.0.0)
   a library which enable you to define abstract method in Ruby

actionmailer (1.3.6, 1.3.5)
   Service layer for easy email delivery and testing.

actionpack (1.13.6, 1.13.5)
   Web-flow and rendering framework putting the VC in MVC.

actionwebservice (1.2.6, 1.2.5)
   Web service support for Action Pack.

activerecord (1.15.6)
   Implements the ActiveRecord pattern for ORM.

activesupport (2.0.2, 1.4.4)
   Support and utility classes used by the Rails framework.

ap4r (0.3.5)
   Asynchr
onous Processing for Ruby.

archive-tar-minitar (0.5.1)
   Provides POSIX tarchive management from Ruby programs.

builder (2.1.2)
   Builders for MarkUp.

cached_model (1.3.1)
   An ActiveRecord abstract model that caches records in memcached

capistrano (2.1.0)
   Capistrano is a utility and framework for executing commands in
   parallel on multiple remote machines, via SSH.

cgi_multipart_eof_fix (2.5.0, 2.3)
   Fix an exploitable bug in CGI multipart parsing.

color (1.4.0)
   Colour management with Ruby

color-tools (1.3.0)
   color-tools provides colour space definition and manpiulation as
   well as commonly named RGB colours.

daemo
ns (1.0.9)
   A toolkit to create and control daemons in different ways

documatic (0.1.0)
   Documatic is an OpenDocument extension for Ruby Reports (Ruport). It
   is a template-driven formatter that can be used to produce
   attractive printable documents such as database reports, invoices,
   letters, faxes and more.

erubis (2.4.1)
   a fast and extensible eRuby implementation which supports
   multi-language

ezcrypto (0.7)
   Simplified encryption library.

fastercsv (1.2.3, 1.2.1)
   FasterCSV is CSV, but faster, smaller, and cleaner.

fastthread (1.0.1)
   Optimized replacement for thread.rb primitives

gem_plugin (0.2
.3)
   A plugin system based on rubygems that uses dependencies only

gruff (0.2.8)
   Beautiful graphs for one or multiple datasets.

highline (1.4.0)
   HighLine is a high-level command-line IO library.

hoe (1.5.0, 1.3.0)
   Hoe is a simple rake/rubygems helper for project Rakefiles

hpricot (0.6)
   a swift, liberal HTML parser with a fantastic library

localization_generator (1.0.8)
   [Rails] Localization generator.

log4r (1.0.5)
   Log4r is a comprehensive and flexible logging library for Ruby.

mailfactory (1.2.3)
   MailFactory is a pure-ruby MIME mail generator

memcache-client (1.5.0)
   A Ruby memcached client

merb (0
.4.1)
   Merb == Mongrel + Erb. Pocket rocket web framework.

mime-types (1.15)
   Manages a MIME Content-Type that will return the Content-Type for a
   given filename.

mongrel (1.1.1)
   A small fast HTTP library and server that runs Rails, Camping, Nitro
   and Iowa apps.

mongrel_cluster (1.0.5)
   Mongrel plugin that provides commands and Capistrano tasks for
   managing multiple Mongrel processes.

needle (1.3.0)
   Needle is a Dependency Injection/Inversion of Control container for
   Ruby. It supports both type-2 (setter) and type-3 (constructor)
   injection. It takes advantage of the dynamic nature of Ruby 20to
   provide a rich and flexible approach to injecting dependencies.

net-sftp (1.1.0)
   Net::SFTP is a pure-Ruby implementation of the SFTP client protocol.

net-ssh (1.1.2)
   Net::SSH is a pure-Ruby implementation of the SSH2 client protocol.

pdf-writer (1.1.7)
   A pure Ruby PDF document creation library.

rails (1.2.6)
   Web-application framework with template engine, control-flow layer,
   and ORM.

rails_analyzer_tools (1.4.0)
   Tools for analyzing the performance of web sites.

railsmachine (1.0.0)
   The Rails Machine task library

rake (0.8.1, 0.7.3)
   Ruby based make-like utility.

RedCloth (3.0.4)
   RedCl
oth is a module for using Textile and Markdown in Ruby. Textile
   and Markdown are text formats. A very simple text format. Another
   stab at making readable text that can be converted to HTML.

reliable-msg (1.1.0)
   Reliable messaging and persistent queues for building asynchronous
   applications in Ruby

rmagick (2.2.2)
   Ruby binding to ImageMagick

rubigen (1.2.3, 1.1.1)
   A framework to allow Ruby applications to generate file/folder stubs
   (like the rails command does for Ruby on Rails, and the
   âscript/generateâ command within a Rails application during
   development).

ruby-debug (0.10.0)
  C2 Command line interface (CLI) for ruby-debug-base

ruby-debug-base (0.10.0, 0.9.3)
   Fast Ruby debugger - core component

ruby-debug-ide (0.1.9)
   IDE interface for ruby-debug.

rubyforge (0.4.4)
   A script which automates a limited set of rubyforge operations.

rubyzip (0.9.1)
   rubyzip is a ruby module for reading and writing zip files

ruport (1.4.0)
   A generalized Ruby report generation and templating engine.

ruport-util (0.8.0)
   A set of tools and helper libs for Ruby Reports

salted_login_generator (1.1.1)
   [Rails] Login generator with salted passwords.

scruffy (0.2.2)
   A powerful, clean graphing library for Ruby.

sources
(0.0.1)
   This package provides download sources for remote gem installation

sql_dep_graph (1.0.0)
   Graphs table dependencies based on usage from SQL logs

SyslogLogger (1.4.0)
   SyslogLogger is a Logger replacement that logs to syslog. It is
   almost drop-in with a few caveats.

tlsmail (0.0.1)
   This library enables pop or smtp via ssl/tls by dynamically
   replacing these classes to these in ruby 1.9.

transaction-simple (1.4.0)
   Simple object transaction support for Ruby.

uuid (1.0.4)
   UUID generator

ZenTest (3.9.1, 3.6.1)
   ZenTest provides 4 different tools and 1 library: zentest,
   unit_diff
, autotest, multiruby, and Test::Rails

regards

Mohammed

On Tue, 26 Feb 2008 03:25:46 -0500
[email protected] wrote:

Hi,

I have been using mongrel + Pound for the past 6 months on a single server deployment.

Recently mongrel has started to behave very weirdly after we?moved app to a bunch of servers from a single server deployment.

These answers might help:

  • It crashes on every exception, throws 500 and throws 500 for ALL instances of the cluster? after one exception has occurred

For the crashes and database problems, make sure you have the mysql or
postgresql gem installed and that it’s using the actual binary library
not the Rails built-in stuff. The built-in driver for those databases
is bad and causes exceptions, lost connections, etc.

  • Connectivity to SQL seems to take a rather long time (the db is a different server on the same network, servers are connected through a gigaport switch)

Rule out your network config before you assume it’s a Rails issue
(not a mongrel issue). Try running a few simple scripts and test the
perf, then start from the most basic network config like half-duplex
and see what you can find.

  • Images deliveries have slowed down since we migrated from single server to multi server deployment (still using pound, not yet moved to apache / nginx)

Yeah, time to upgrade to nginx. Nginx is actually easier to configure
than pound, makes more sense in the config, and is a hell of a lot
faster and stable. Especially for having it serve static files. Just
drop pound.

  • The application architecture uses lot of background processes?- built using?DRB which again are deployed physically on different dedicated machines to distribute load

Yeah, this really makes me wonder what you’re doing. I’d go and double
and triple check every line of code and watch network traffic to make
sure you aren’t doing something dumb like sending huge files over DRb.
I’m guessing you’ve got something dumb that’s overloading the Ruby
interpreter.

  • ?Mongrel doesn’t always log the error 500 within its logs (mongrel may not be getting these requests).

Mongrel doesn’t log most stuff, rails does. Check in the
production.log file and see what’s there.

-? Other mongrel instances in the same cluster do not respond even if one mongrel has crashed or has encountered an exception.

That sounds like Pound more than anything. Isolate the difference
between a connection to pound and one to mongrel.

Here is some basic info of the deployment

With this much hardware you should be able to crank tons of requests
through the system. If you have this many problems, it’s entirely in
your design. Remember, tons and tons of other people do way more with
much less.


Zed A. Shaw

Hi~

On Feb 26, 2008, at 2:44 AM, [email protected] wrote:

Dave,

Here is the version of Ruby along with other Gems installed

ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]

This version of ruby has serious bugs! You need to either down grade
to ruby 1.8.5 or upgrade to ruby 1.8.6 p 111. You need at least patch
level 110 or else do not use 1.8.6

Cheers-

On 26 Feb 2008, at 19:53, Ezra Z. wrote:

ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux]

This version of ruby has serious bugs! You need to either down grade
to ruby 1.8.5 or upgrade to ruby 1.8.6 p 111. You need at least patch
level 110 or else do not use 1.8.6

Thanks for that tip – I checked my Ruby installation and found it
was that very same buggy version. I hadn’t realised it was flawed…

Regards,
Andy S.

Zed, Emobius,

Thank you for the replies.

Here’s what I found - mysql gem was not installed and that maybe a core
source of problems as suggested by Zed. We’ve had? too many problems
when we tried to install this, and found that none of the solutions that
are mentioned really work. We will try to install this gem again.

An application that we built using DRB designed to be asynchronous
failed to process asynchronous requests, had each instance of ruby
piling up request queues and subsequently throwing errors (as all ports
were getting requests continuously).

I will be keen to get inputs from anyone who has used DRB instead of
AP4R for asynchronous requests

Ruby version is 1.8.6 (we’re upgrading this today night)

Ngnix - we’re trying this out today on our staging machines and after it
is found to be stable, we will move to production.

Temporarily, we’ve moved from DRB to AP4R for asynchronous processing.
Personally, I am still comfortable with DRB than AP4R but that is
another discussion altogether.


Another question: Are there benchmarked transaction statistics on
standard hardware for ROR / mongrel platform (similar to whats available
on TPC.org?). We’re looking at benchmarking our system to initially
support 10m requests per day (consider each one to be a simple
find_by_sql query against a database of 1000 requests + 1 asynchronous
process for billing.

regards

Mohammed. A

To install mysql gem you need to specify the mysql_config path(if it is
not
installed then into mysql dev or devel package)
To install sudo gem install mysql –
–with-mysql-config=<path-to-mysql_config like on my ubuntu it is
/usr/bin/mysql_config>

And your problems are similar(not same) I used to encounter. I had these
problems with my mongrel handler when it used to write to database in
asynchronous queries by the users. Log file used to scream connection
lost
to mysql. After spending a few days trying to boost up my mysql server I
did
the gem install and since then it has handled 100K + queries (the
mongrel
handler part of the app)each day without any problem whatsoever.

Also switch to apache/ngnix if you can. Pound could also be the trouble
maker though I am not very sure about this comment.

I hacked our vendor tree to explode on startup if the mysql gem could
not be found. IMHO this should be the default behavior, the built in
mysql adapter is crap and its silent use is a ticking time bomb

Cheers

Dave

Switching to NGINX indeed fixed much of the problems. But mysql gem
continues to evade no matter what kind of config I use.
But SQL has been tuned up, major bugs have been tuned.

Much better now. Will post later with detailed analysis of the problem
and solutions

regards

Mohammed. A