POST with huge HTTP body

Hi,

I found the following page describing how to stream data from the
server to the client:

http://api.rubyonrails.com/classes/ActionController/Streaming.html

I want to do the opposite, streaming data from the client to the server,
letting the controller saving the data while it’s being received.
In particular, I want to upload large files to the RailsDav
(http://www.liverail.net/railsdav) plugin, without having to store
the entire file in memory during the operation.

This seems to be a Mongrel problem, so I started looking at this plugin:

http://mongrel.rubyforge.org/docs/upload_progress.html

How do I get Mongrel to store the incoming POST data in a tempfile,
passing a handler to that file as the raw_post field in the Rails
request?
Is it even possible?
How is the “request” parameter to the process() method related to the
“request” object in the Rails controllers?

/Daniel

On 10/15/07, Daniel B. [email protected] wrote:

I want to do the opposite, streaming data from the client to the server,
letting the controller saving the data while it’s being received.
In particular, I want to upload large files to the RailsDav
(http://www.liverail.net/railsdav) plugin, without having to store
the entire file in memory during the operation.

This seems to be a Mongrel problem, […]

I don’t have the solution to your problem, but I think you are looking
in
the wrong place.

Unless you plan to have mongrel listening on a public port, this is not
a
mongrel problem. The standard way to use mongrel, with Rails or not, is
to
set up a few of them behind a reverse proxy like nginx or
apache+mod_proxy_balancer. In those cases, the proxy will not forward
the
data to mongrel before it has received it all, and once it has all the
data,
the latency between nginx and mongrel is negligible.

On the other hand, the reverse proxy (I’m only speaking from experience
with
nginx here) saves the data to file while it is receiving it. I think
you
should look for a solution that lets nginx pass the path to that file to
mongrel rather than sending the data again. I believe I have read about
such a solution on this very list a few months ago.

So, not the help you wanted, but I hope it leads you in the right
direction. Well, if you actually intend to have mongrel face the
public,
then just forget what I said.

Good luck!

/David

At Mon, 15 Oct 2007 14:00:20 +0200,
“Daniel B.” [email protected] wrote:

received. In particular, I want to upload large files to the
RailsDav (http://www.liverail.net/railsdav) plugin, without having
to store the entire file in memory during the operation.

This seems to be a Mongrel problem, so I started looking at this plugin:

http://mongrel.rubyforge.org/docs/upload_progress.html

How do I get Mongrel to store the incoming POST data in a tempfile,
passing a handler to that file as the raw_post field in the Rails request?
Is it even possible?

Hi Daniel,

Mongrel is going to put a large POSTed body into a tempfile. Examine
the mongrel.rb file around line 220. In this case your request.body is
going to be a File object. Why do you think that there is a mongrel
problem here? See also what Tim K. said in this thread.

How is the “request” parameter to the process() method related to
the “request” object in the Rails controllers?

request in mongrel (as sent to process) is an instance of
Mongrel::HttpRequest. In Rails it is an instance of
ActionController::CgiRequest.

best,
Erik Hetzner
;; Erik Hetzner, California Digital Library
;; gnupg key id: 1024D/01DB07E3