I have a setup with apache 2.2.6 proxying requests to a cluster of 5
mongrels. Occassionally, I receive the following error in the logs:
[Tue Oct 30 12:00:28 2007] [error] (70007)The timeout specified has
expired: proxy: pass request body failed to 127.0.0.1:8013 (127.0.0.1)
from 213.205.247.171 ()
[Tue Oct 30 12:00:28 2007] [error] [client 213.205.247.171] Handler for
proxy-server returned invalid result code 70007
It may be worth noting that the Mongrels are dealing with potentially
long running HTTP POSTs: they come from Mobile Clients uploading around
2 -3 MB of data, which can take several minutes. It is my understanding
that a Mongrel will not be tied up whilst the upload is being done as
Mongrel will create a thread to handle the upload, stream it to a
TempFile, and then lock rails. The rails action that deals with the
file typically takes 1/2 second to 1 second. Looking at the error
message, it looks like Mongrel may be ignoring the request or Apache is
Timing Out whilst forwarding the request to Mongrel.
Can anyone help in decrypting what this error message may mean?
I believe that this error is reported when the client stops sending
data. If you grep through the sources the area that generates this
error is still related to the request. For us, this was a clue that
eventually pointed to a duplex mismatch upstream from our rack that
was causing tcp connections to collapse under heavy traffic.
Does this mean that when a request comes in, Apache doesn’t hand off the
request to Mongrel until it has received the entirety of the request?
If not, how does Apache know the connection has been closed/client has
stopped sending data.
I thought it worked (roughly) like:
1 Apache receives beginning of request (Content-length headers and so
on)
2 Apache forwards request to Mongrel
3 Mongrel begins reading request (in my case uploading a 3 MB file over
a slow connection)
In the above scenario, if the client stopped sending data, would it not
be Mongrel that raised the error?
Thanks very much!
Dave C. wrote:
Hi Matt,
I believe that this error is reported when the client stops sending
data. If you grep through the sources the area that generates this
error is still related to the request. For us, this was a clue that
eventually pointed to a duplex mismatch upstream from our rack that
was causing tcp connections to collapse under heavy traffic.
hi,
i am getting this error as well but my process is a small one. it seems
after my rails app is running for a day, i notice that every so
often(every 3-4 clicks on the site), the process hangs. after i restart
my mongrels, its back to normal again. any solution you have found?
If your database is MySQL, check to see you have the C-based MySQL
driver - it is usually installed as a gem: ‘gem install mysql’
Installing that seems to have cured my problems with my development
servers dying when left alone overnight.
It isn’t an explanation or even a fix - but setting up something that
monitors your web site by making an http request for a simple page - I
have a Rails controller with an action that just renders the text
“success” - will hide the issue. For example, nagios monitoring might
keep your servers running. If not, this is definitely not an
inactivity issue but something else that causes your mongrels to die.
If your database is MySQL, check to see you have the C-based MySQL
driver - it is usually installed as a gem: ‘gem install mysql’
Installing that seems to have cured my problems with my development
servers dying when left alone overnight.