Nginx Upload Progress Module stops functioning once behind reverse-proxy

Hello,

I’ve recently compiled nginx-1.7.10 with a third-party upload-progress
tracking module, which is described at
NGINX Upload Progress Module | NGINX .

This module works perfectly well, until I attempt to put the entire
setup behind a reverse-proxy.

Below is a simplified configuration that demonstrates the issue I’m
having.

With this configuration, the upload progress reporting functions
correctly only if I access the reverse-proxy port directly, at
https://example.com:1337.

As soon as I attempt to access the upstream host at https://example.com
(on port 443, instead of the proxy port, 1337), the upload progress
module always reports “starting” as its state (which, according to the
documentation, implies “the upload request hasn’t been registered yet or
is unknown”).

Once I have this working with the reverse-proxy in-place, I want to
change the line “listen *:1337 ssl;” to “listen 127.0.0.1:1337 ssl;” so
that user-agents cannot access the proxy directly.

Might anyone know why this module ceases to work correctly once behind a
reverse-proxy, as the below config demonstrates?

Is there some proxy-specific configuration directive that I’ve
overlooked?

Thanks for any help!

-Ben

###################################################################
http {
server {
listen *:443 ssl;

location ^~ / {
  proxy_pass https://127.0.0.1:1337;
}

}

upload_progress uploads 5m;

server {
listen *:1337 ssl;

location @virtual {
  include fastcgi_params;
  fastcgi_pass unix:/var/lib/php5-fpm/web1.sock;
  fastcgi_index index.php;
  fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
  fastcgi_intercept_errors on;
}

location = /progress/ {
  report_uploads uploads;
  upload_progress_json_output;
}

location /file-upload/progress/ {
  proxy_pass https://127.0.0.1:1337;
  track_uploads uploads 5s;
}

}
}
###################################################################

On Mon, Feb 16, 2015 at 02:55:51PM -0500, Ben J. wrote:

Hi there,

I’ve recently compiled nginx-1.7.10 with a third-party upload-progress
tracking module, which is described at
NGINX Upload Progress Module | NGINX .

Read the first two paragraphs on that page…

This module works perfectly well, until I attempt to put the entire
setup behind a reverse-proxy.

…and draw a picture of what happens when a client sends a 1MB upload
to nginx, and nginx sends it via proxy_pass to an upstream. That should
show you why it fails.

With this configuration, the upload progress reporting functions
correctly only if I access the reverse-proxy port directly, at
https://example.com:1337.

Yes.

That’s what nginx does.

Why does there need to be an upload_progress module in the first place,
instead of letting the upstream server take care of it?

In this case, you have put your upload_progress module on your upstream
server.

Not going to work. At least until you can use an “unbuffered
upload”. Which is not in current nginx. (But is in other things.)

f

Francis D. [email protected]

On 2/16/2015 3:52 PM, Francis D. wrote:

On Mon, Feb 16, 2015 at 02:55:51PM -0500, Ben J. wrote:

Hi there,

Hi, Francis, and thank you for taking the time to review my question and
respond. I value and appreciate your time.

to nginx, and nginx sends it via proxy_pass to an upstream. That should
show you why it fails.

Abashedly, as yet, I lack the familiarity with nginx (and
reverse-proxies, in general) to grasp the problem on a fundamental
level. I’ve tried to visualize the conundrum and I can see two potential
problems:

  • The fact that this is really a three-tiered server setup: 1) the nginx
    server running on port 443; 2) the nginx server running on port 1337; 3)
    the PHP server running on the socket.

  • The fact that in the config snippet I included, the server listening
    on port 1337 is “feeding the file to itself”. Intuitively (and
    incorrectly, of course), it feels as though changing that second
    proxy_pass to

proxy_pass https://127.0.0.1;

would solve the problem. But, as you suggested, it does not and cannot.

Any further explanation that you’re willing to offer as to the
fundamental problem will not fall on deaf ears.

Because the upstream server (PHP) performs miserably by comparison when
handling massive file uploads (often in excess of 1GB), and doesn’t
support the most important (to me) features of this nginx module,
coupled with the nginx Upload Module (
http://www.grid.net.ru/nginx/upload.en.html ).

In particular, I need the ability to track upload progress, server-side,
and support user-agents resuming uploads that fail for any reason.

In this case, you have put your upload_progress module on your upstream
server.

That’s fine. I was hoping to avoid it, but I have control over the
entire environment, so it’s not a show-stopper.

I was hoping to implement the reverse-proxy configuration so that the
file-hungry virtual-host would have minimal potential to affect the
handful of other virtual-hosts running under the same nginx instance.

The idea was to “isolate” the virtual-host that employs these modules so
that it could be restarted, be replaced with a more recent version of
nginx, etc., with no real downtime risk to the upstream nginx instance.

Imagine a hosting environment in which the “top-most” (furthest
upstream) virtual-hosts are critical and should never drop offline for
any period of time, but wherein it’s no big deal if the “child” nginx
instances go down for whatever reason. This was the topology I was
hoping to implement.

Not going to work. At least until you can use an “unbuffered
upload”. Which is not in current nginx. (But is in other things.)

f

I would rather push this functionality upstream and consolidate all
virtual-hosts, incurring whatever risk of downtime is introduced as a
result, than replace nginx with another piece of software.

Thanks again for assuring me that this simply won’t work as-is and that
the best and only course of action is to move the virtual-host in
question upstream.

Eager to hear any additional thoughts or critiques you may have re: the
justification that I outlined. Maybe I’m “doing it wrong”.

Respectfully,

-Ben

On Tue, Feb 17, 2015 at 09:30:46PM -0500, Ben J. wrote:

On 2/16/2015 3:52 PM, Francis D. wrote:

Hi there,

…and draw a picture of what happens when a client sends a 1MB upload
to nginx, and nginx sends it via proxy_pass to an upstream. That should
show you why it fails.

Any further explanation that you’re willing to offer as to the
fundamental problem will not fall on deaf ears.

nginx buffers the entire request before sending anything to upstream.

So upstream sees nothing while the client is sending, and then sees
everything, fed by the (presumably) fast link between it and nginx.

So the only place a progress meter can usefully be is on the initial
nginx, which is talking directly with the client. Anything upstream of
that will show 0% until it shows 100%, which is not really a “progress”
meter.

Why does there need to be an upload_progress module in the first place,
instead of letting the upstream server take care of it?

Because the upstream server (PHP) performs miserably by comparison when
handling massive file uploads (often in excess of 1GB),

Nope. At least, that’s not the main reason. The upstream server does
not see the progress, and so cannot report on it.

Not going to work. At least until you can use an “unbuffered
upload”. Which is not in current nginx. (But is in other things.)

I would rather push this functionality upstream and consolidate all
virtual-hosts, incurring whatever risk of downtime is introduced as a
result, than replace nginx with another piece of software.

Search for “nginx unbuffered upload”.

You can possibly try patching yourself, or wait until stock nginx
supports
it, or offer suitable encouragement to get stock nginx to support it on
your preferred timetable.

But until it exists, only the thing talking to the client can offer any
kind of progress meter.

Cheers,

f

Francis D. [email protected]