Mongrel goes unresponsive after 30 minutes

Greetings!

I’m in the process of moving my app from a shared hosting arrangement to
a
VPS and am seeing some mongrel behavior I haven’t seen before.
Specifically, after about 30 minutes of inactivity mongrel goes
unresponsive
and has to be restarted. When I restart it writes the messages below.
Before that, there’s nothing in the logs that shows any problem at all.

In mongrel.log:

** USR2 signal received.
Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads for slow workers because
of
‘shutdown’
Thread # is too old, killing.
Thu Feb 08 08:43:51 EST 2007: Error calling Dispatcher.dispatch #Thread

is

too old, killing.

In production.log:

Mongrel timed out this thread: shutdown

A set of trace messages follow this which I’m happy to post if it would
be
helpful.

Can anybody help me understand what’s going on and what I need to do to
fix
it?

Thanks much,
Bill

From Q: Mongrel stops working if it’s left alone for a long time.

If you find that Mongrel stops working after a long idle time and
you’re using MySQL then you’re hitting a bug in the MySQL driver that
doesn’t properly timeout connections. What happens is the MySQL server
side of the connection times out and closes, but the MySQL client
doesn’t detect this and just sits there.

What you have to do is set:

ActiveRecord::Base.verification_timeout = 14400

Or to any value that is lower than the MySQL server’s
interactive_timeout setting. This will make sure that ActiveRecord
checks the connection often enough to reset the connection.

Q: Mongrel stops working if it’s left alone for a long time.

If you find that Mongrel stops working after a long idle time and
you’re using MySQL then you’re hitting a bug in the MySQL driver that
doesn’t properly timeout connections. What happens is the MySQL server
side of the connection times out and closes, but the MySQL client
doesn’t detect this and just sits there.

What you have to do is set:

ActiveRecord::Base.verification_timeout = 14400

Or to any value that is lower than the MySQL server’s
interactive_timeout setting. This will make sure that ActiveRecord
checks the connection often enough to reset the connection.


Aníbal Rojas

Hi Anibal,

Thanks! I’ve got the admin looking into now. I really appreciate the
response.

Best regards,
Bill

The previous snippet is from:

http://mongrel.rubyforge.org/faq.html


Aníbal Rojas

Bill W. wrote:

** USR2 signal received.
A set of trace messages follow this which I’m happy to post if it would be
helpful.

Can anybody help me understand what’s going on and what I need to do to fix
it?

Do you have a cluster of mongrels running or just the one?

-Robby


Robby R.
http://www.robbyonrails.com/

Hi Robby,

Robby R. wrote:

Do you have a cluster of mongrels running or just the one?

I’m not sure and asked similar question of my hosting service today.
I’ll
report back on that as soon as I find out. For development purposes I
started with a shared hosting plan with the same provider and the app
I’m
having problems with is running just fine on that site. I haven’t even
browsed to it in almost two weeks and when I did this afternoon I found
that
it’s just ‘hummin along’. It’s not _exactly_the same app, but there are
very few differences. My first instinct is to look for differences in
the
code between the two but, IME, that’s really only going to be a
productive
approach if the underlying architecture is comparable. Any debugging
advice
/ experience is tremendously appreciated.

Thanks!
Bill

You are welcome, good luck. Zed has a checklist in Mongrel’s site for
some other conditions that could cause this errors.


Aníbal Rojas

On Thu, 8 Feb 2007 09:07:47 -0600
“Bill W.” [email protected] wrote:

Greetings!

I’m in the process of moving my app from a shared hosting arrangement to a
VPS and am seeing some mongrel behavior I haven’t seen before.
Specifically, after about 30 minutes of inactivity mongrel goes unresponsive
and has to be restarted. When I restart it writes the messages below.
Before that, there’s nothing in the logs that shows any problem at all.

Restart (USR2) isn’t that reliable because of how ruby/rails needs to
reload. You should use a full stop/start cycle instead.

Once you start doing that and you still get no response then run strace
on the mongrel process to see what it’s doing. It’s most likely that
you have a library that’s messing things up.

Also, don’t use pstore, don’t set Logger to rotate, don’t do file
locking, and don’t use a crappy DNS.


Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu

http://www.awprofessional.com/title/0321483502 – The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 – Come get help.

Hi Zed,

Thanks very much for your reply. I appreciate it very much. More
below.

Zed A. Shaw wrote:

Restart (USR2) isn’t that reliable because of
how ruby/rails needs to reload. You should
use a full stop/start cycle instead.

I’ve been doing that. Should have said it that way.

Once you start doing that and you still get no
response then run strace on the mongrel process
to see what it’s doing.

The folks at the hosting service (a2hosting) have done that I think and
they
tell me it hasn’t given them any clues.

It’s most likely that you have a library that’s messing
things up.

If by ‘library’ you mean gems / plugins, I do have a couple installed:
PDF::Writer and BackgroundRb. But BackgroundRb is stopped and there are
no
calls to PDF:Writer being used in the test scenerio I’ve been using. I
thought Rails didn’t load code before it needed it. Anyway, here’s what
I’m
doing.

  1. Start mongrel using mongrel_rails start -d -p 3002 -e production <
    /dev/null >& /dev/null
  2. Open a browser and browse to the index action of the Rails app (the
    index
    action is empty and the view renders a page with a form on it that has a
    group of 3 radio buttons and a submit button.)
  3. Close the browser.
  4. Wait 40 minutes, then do step 2 again.

The result of step 4 is a blank white screen, not an error message.
Neither
production.log nor mongel.log have anything written to them at that
point
other than the initial startup messages in mongrel.log and the initial
request from step 2 in production.log. When I stop mongrel, it writes a
message to both logs saying its killing a slow worker. That’s the only
clue
I get. Steps 1-4 can be repeated ad nauseum.

After their initial investigation the sys admins concluded the problem
had
to be in my code (surprise, surprise :wink: ) I had a slightly different
version that was running flawlessly on my shared account with them, so I
decided to ‘dig in.’ I copied the entire Rails application directory
for
both apps to my PC, then copied the one I was having problems with on
the
VPS over to the shared account. It ran without a problem (and, a week
later, is still running without a problem.)

Yesterday they set up a cron job outside my space to invoke the index
method
every 10 minutes. That’s ‘solved’ the problem. It doesn’t sit well
with me
though. My experience leads me to fear that whatever the real cause of
the
problem is will eventually manifest itself again. At the worst possible
time.

If you (or anyone else reading this) have any interest in understanding
what’s really going on (since mongrel’s behavior is the only clue from
an
end user perspective), please let me know. I’ll give you any access you
need. If not, I understand completely.

Thanks again for your time.

Best regards,
Bill

On Fri, 16 Feb 2007 09:20:11 -0600
“Bill W.” [email protected] wrote:

Yesterday they set up a cron job outside my space to invoke the index method
every 10 minutes. That’s ‘solved’ the problem. It doesn’t sit well with me
though. My experience leads me to fear that whatever the real cause of the
problem is will eventually manifest itself again. At the worst possible
time.

Interesting, make sure you’ve got the compiled mysql gem and you aren’t
using the rails mysql.rb file.

If you (or anyone else reading this) have any interest in understanding
what’s really going on (since mongrel’s behavior is the only clue from an
end user perspective), please let me know. I’ll give you any access you
need. If not, I understand completely.

Yeah, actually I’d be interested. Email me off list to coordinate it.


Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu

http://www.awprofessional.com/title/0321483502 – The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 – Come get help.

On 2/16/07, Zed A. Shaw [email protected] wrote:

make sure you’ve got the compiled mysql gem and you aren’t
using the rails mysql.rb file.

Zed,

Can you point me to the reason for this recommendation? Is there a known
problem with pure-Ruby MySQL driver?

Best regards,
Alexey V.

The problem has been resolved. I’ve included the ‘bread crumbs’ below
to
both recognize the contributors and to document it for future Railers.
Thank you to all.

Best regards,
Bill

Bill W. wrote:

I’m in the process of moving my app from a shared
hosting arrangement to a VPS and am seeing some
mongrel behavior I haven’t seen before. Specifically,
after about 30 minutes of inactivity mongrel goes
unresponsive and has to be restarted.

Zed A. Shaw wrote:

make sure you’ve got the compiled mysql gem and
you aren’t using the rails mysql.rb file.

Ezra Z. wrote:

Also if your host is running linux then you can tell for
sure if it’s the compiled driver from irb:

ey00-s00059 ~ # irb
irb(main):001:0> require ‘mysql.so’
=> true

Bill W. wrote:

irb(main):002:0>
Aníbal Rojas wrote:
As I undertand the ruby-mysql library 2.7 does
require the libmysqlclient to be properly installed,
and depending on the location of mysql.so it is
possible that rubygems is unable to load it, while
the library is actually using it…

Support at hosting service wrote:

This issue has been rectified:

[email protected] [~]# irb
irb(main):001:0> require ‘mysql.so’
=> true
irb(main):002:0>

And that fixed the behavior mongrel was exhibiting.