Nginx unexpectedly returning 304 for content cached from proxy

Server NGINX
1

Hello!
We recently started using Nginx 0.7.59 to replace Squid on a couple of
our
static file caching servers (Debian 5.0). However, nginx is returning
“HTTP
304 Not Modified” responses for some—not all—of the cached images, even
if
the browser doesn’t send an If-modified-since header – causing lots of
broken images when viewing the site in browser.

As soon as I turn off the “proxy_cache” directive and let the requests
be
proxied to the upstream server, bypassing nginx’s cache, no more
unexpected
304’s are served, and the broken images disappear.

Do you have any idea of what might be causing this problem?

My nginx.conf file is below:

worker_processes 3;

error_log logs/error.log;

events {
worker_connections 10240;
use epoll;
}

http {
include mime.types;
default_type application/octet-stream;

access_log  off;

sendfile        on;
keepalive_timeout  10;

gzip  on;

upstream backend {
    server  xxx.xxx.xxx.xxx;
}

proxy_buffer_size 16k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 32k;
proxy_intercept_errors on;
proxy_cache_path /var/www/cache levels=1:2
        keys_zone=zone_one:10m inactive=24h;

server {
    listen       80;
    server_name  localhost;

    location / {
        proxy_cache zone_one;
        proxy_cache_valid 200 7d;
        proxy_pass http://backend;
    }
}

}

I added the “proxy_cache_valid” directive to only cache 200 responses,
but
that didn’t seem to change anything (even after deleting the cache).

This is pretty much what I’m seeing:

$ curl -I
http://www.myserver.com/thumbs/125x100/10081/m88_1240816236.jpg
HTTP/1.1 304 Not Modified
Server: nginx/0.7.59
Date: Mon, 08 Jun 2009 09:26:05 GMT
Connection: keep-alive
Last-Modified: Mon, 27 Apr 2009 07:10:40 GMT
Expires: Wed, 08 Jul 2009 08:19:13 GMT
Cache-Control: max-age=2592000

If I disable the proxy_cache (leaving just proxy_pass in location /):

$ curl -I
http://www.myserver.com/thumbs/125x100/10081/m88_1240816236.jpg
HTTP/1.1 200 OK
Server: nginx/0.7.59
Date: Mon, 08 Jun 2009 09:28:17 GMT
Content-Type: image/jpeg
Connection: keep-alive
Content-Length: 3869
Last-Modified: Mon, 27 Apr 2009 07:10:40 GMT
Vary: Accept-Encoding
Expires: Wed, 08 Jul 2009 09:32:33 GMT
Cache-Control: max-age=2592000
Accept-Ranges: bytes

I’m not quite sure where to start debugging this. Does anyone have any
advice?

Thanks,
Antonio

2

Please don’t double post to the list and the forum.

Thanks

3

On Mon, Jun 08, 2009 at 01:39:12PM +0400, Antonio L. wrote:

worker_connections  10240;
keepalive_timeout  10;
proxy_intercept_errors on;
        proxy_pass http://backend;
    }
}

}

This is a bug. I will try to fix in today 0.8.1.
As quick workaround, you may set any proxy_set_header directive in
cached location, e.g.:

     location / {
         proxy_cache zone_one;
         proxy_cache_valid 200 7d;
         proxy_pass http://backend;

  •        proxy_set_header  Host  backend;
   }
4

Thank you! Adding the proxy_set_header directive worked (after clearing
the
cache).
I apologize for the double post.

-a

5

I’m getting similar problems with 304 messages being incorrectly
returned with nginx 0.8.14:

HEAD /css/rich.css HTTP/1.1
User-Agent: curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8i zlib/1.2.3 libssh2/0.15-CVS
Host: prod-login.uqconnect.net
Accept: /

< HTTP/1.1 304 Not Modified
< Server: nginx/0.8.14
< Date: Mon, 21 Sep 2009 04:52:33 GMT
< Connection: keep-alive
< ETag: “189ad-1b68-4727c27516700”
< Expires: Mon, 21 Sep 2009 04:52:34 GMT
< Cache-Control: max-age=5

instead of:

HEAD /css/rich.css HTTP/1.1
User-Agent: curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8i zlib/1.2.3 libssh2/0.15-CVS
Host: prod-login.uqconnect.net
Accept: /

< HTTP/1.1 200 OK
< Server: nginx/0.8.14
< Date: Mon, 21 Sep 2009 04:52:32 GMT
< Content-Type: text/css
< Connection: keep-alive
< Last-Modified: Tue, 01 Sep 2009 03:57:45 GMT
< ETag: “20c55-1b68-4727c2723a040”
< Content-Length: 7016
< Cache-Control: max-age=5
< Expires: Mon, 21 Sep 2009 04:52:33 GMT
< Accept-Ranges: bytes

This only happens occasionally. Certainly less than 10% of requests.
Unfortunately, failing to load CSS is rather spectacular…

The relevant portion of my config is:

gzip on;
gzip_proxied any;

Need Vary on encoding though, otherwise caches may get confused

gzip_vary on;

error_log /var/log/nginx/error.log info;

proxy_temp_path /var/tmp/nginx/tmp;

Should not be used, but useful for ensuring “cache” dir is created

proxy_cache_path /var/tmp/nginx/cache levels=1:2
keys_zone=core:1m;

Actual caches start here

proxy_cache_path /var/tmp/nginx/cache/itmp-prod levels=1:2
keys_zone=itmp-prod:1m;
proxy_cache_path /var/tmp/nginx/cache/itmp-test levels=1:2
keys_zone=itmp-test:1m;

proxy_cache_valid 200 302 5m;
proxy_cache_valid 301 1h;
proxy_cache_valid 404 30s;

upstream apache {
server apache.localhost:8080;
server itmp1-gpn.soe.uq.edu.au:8080 backup;
server itmp1-prentice.soe.uq.edu.au:8080 backup;
server itmp2-prentice.soe.uq.edu.au:8080 backup;
}

server {
listen 80;
server_name prod-login.uqconnect.net;

    proxy_cache itmp-prod;

    location / {
            # Proxy to apache backends
            proxy_pass http://apache;
            # Set header to be what we requested
            proxy_set_header        Host    $host;
            # Need this for snooping with tcpdump (turns off

upstream compression)
proxy_set_header Accept-Encoding “”;
# Set real IP header (needed for portal client IP
detection)
proxy_set_header X-Real-IP $remote_addr;
}

}

server {
listen 80;
server_name test-login.uqconnect.net;
server_name dev-login.uqconnect.net;

    proxy_cache itmp-test;

    location / {
            # Proxy to apache backends
            proxy_pass http://apache;
            # Set header to be what we requested
            proxy_set_header        Host    $host;
            # Need this for snooping with tcpdump (turns off

upstream compression)
proxy_set_header Accept-Encoding “”;
# Set real IP header (needed for portal client IP
detection)
proxy_set_header X-Real-IP $remote_addr;
}

}

Any ideas on a work-around or further debugging details I should
provide?

Thank you,

Tim

Posted at Nginx Forum:

6

the similar issue:
if use default setting,If-Modified-Since header is not fw to back end,
when the cache file expired the cache server will get the file again
from the backend server even if the file is not changed,it’s not good
for a high traffic load site.
is use proxy_set_header If-Modified-Since $http_if_modified_since,then
will get unexpectedly returning 304 .
cache server cache 304 is not good i think…
Maybe the nginx should see wether the cache file exist first:
if exist then fw the if-Modified-Since header to backend,if the file has
been changed, get file from the backend and overwrite the cache file,if
not changes,return 304,then remain the cache file.
if no exist,then get rid of the If-Modified-Since header to get a new
file from the backend

Posted at Nginx Forum:

7

I believe I have now located cause of this problem. It appears that
“If-None-Match” is being passed back to the Apache backend. (For
reference: If-None-Match
HTTP/1.1: Header Field Definitions)

Unfortunately, the 0.8.1 fix doesn’t apply to this header:

*) Bugfix: the “If-Modified-Since”, “If-Range”, etc. client request
header lines were passed to backend while caching if no
“proxy_set_header” directive was used with any parameters.

As a result, the Apache backend can still return a 304, which nginx
happily caches. This is obviously not correct behavior for HTTP/1.1
(RFC2616):

If a 304 response indicates an entity not currently cached, then the
cache MUST disregard the response and repeat the request without the
conditional.

As a workaround, I’m removing this header for proxying:

proxy_set_header If-None-Match “”;

However, this causes requests using only If-None-Match to return 200 OK:

GET /css/rich.css HTTP/1.1
User-Agent: curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3 OpenSSL/0.9.8k zlib/1.2.3 libssh2/0.15-CVS
Host: prod-login.uqconnect.net:444
Accept: /
If-None-Match: “48892-1dfa-47682392e3c00”

< HTTP/1.1 200 OK
< Server: nginx/0.8.17
< Date: Tue, 27 Oct 2009 01:28:41 GMT
< Content-Type: text/css
< Connection: keep-alive
< Last-Modified: Thu, 22 Oct 2009 08:53:04 GMT
< ETag: “48892-1dfa-47682392e3c00”
< Accept-Ranges: bytes
< Content-Length: 7674
< Cache-Control: max-age=5
< Expires: Tue, 27 Oct 2009 01:28:46 GMT

This is fine for now, though not ideal. RFC2616 only states that a
server SHOULD return a 304, not that it must. (
HTTP/1.1: Header Field Definitions ).

I don’t think the work-around should be required though. While the proxy
module is limited to HTTP/1.0 (and RFC1945 doesn’t include
“If-None-Match”), nginx should still comply with RFC2616 for its own
replies without extra configuration. This bug is particularly nasty
because the current stripping of “If-Modified-Since” means that 304 will
be returned in cases where the original preconditions would have
generated a 200 OK.

Any chance of a bug fix?

Thank you,

Tim

Posted at Nginx Forum: