Nginx cache files by mime type using ngx_srcache module

alcina · June 11, 2013, 11:11am

Hi,

I’m using ngx_srcache module with Memcached to cache static contents of
the
upstream servers. This is based on static file extensions. Now I need to
cached them by looking at Content-Type of the header to cache .html
pages as
we do not use the .html extension. (i.e.
http://www.example.com/this-is-my-site )

Is this possible with ngx_srcache module using map module or something
similar…

http {
map $upstream_http_content_type $no_proxy {
default 0;
~*^html/ 0;
}

and hook this $no_proxy to ngx_srcache variables like I’m doing with
extension…?

location ~* \.(html)$ {
                    set $key  $uri$args;
                    srcache_fetch GET /memc key=$key;
                    srcache_store PUT /memc

key=$key&exptime=$ttl_1m;
proxy_pass http://www.example.com;
}

Any help to this is really appreciated.

Thanks in advance.

Posted at Nginx Forum:

n1xman · June 11, 2013, 9:06pm

Hello!

On Tue, Jun 11, 2013 at 2:10 AM, n1xman wrote:

default 0;
~*^html/ 0;
}

and hook this $no_proxy to ngx_srcache variables like I’m doing with
extension…?
location ~* \.(html)$ {
                    set $key  $uri$args;

Just a side note: because you’re using the key in the “key” URI
argument below, you should escape it because the value may contain
special characters. That is,

set_escape_uri $key  $uri$args;

And then in your location = /memc, you should unescape the “key” URI
argument like this:

set_unescape_uri $key  $arg_key;

                    srcache_fetch GET /memc key=$key;
                    srcache_store PUT /memc key=$key&exptime=$ttl_1m;
                    proxy_pass http://www.example.com;
            }

You can use the srcache_store_skip directive to achieve this:

http://wiki.nginx.org/HttpSRCacheModule#srcache_store_skip

Basically, you can do something like this:

map $upstream_http_content_type $no_store {
    default 1;
    ~*html  0;
}

server {
    ...
    location / {
       set_escape_uri $key $uri$args;
       srcache_fetch GET /memc key=$key;
       srcache_store PUT /memc key=$key&exptime=$ttl_1m;
       proxy_pass http://www.example.com;

       srcache_store_skip $no_store;
   }
}

Best regards,
-agentzh

n1xman · June 12, 2013, 12:51pm

Hi agentzh,

Thanks for the support and it is working

However, I have noticed every srcache_fetch GET /memc subrequest hits
the
memcached even though srcache_store skipped content-type which is not
defined.

This will keep busy the memcached server and I think srcache_fetch GET
subrequest should not executed if srcache_store_skip triggered first. Or
am
I missing something here…

map $upstream_http_content_type $no_store {
default 1;
~*html 0;
}

server {
…

location /memc {
            internal;
            set_unescape_uri $memc_key $arg_key;
            set $memc_key $arg_key;
            set $memc_exptime $arg_exptime;
            memc_pass 127.0.0.1:11211;
}

            location / {
                    srcache_response_cache_control off;
                    set_escape_uri $key $uri$args;
                    set $ttl_1d 60;
                    srcache_fetch GET /memc key=$key;
                    srcache_store PUT /memc

key=$key&exptime=$ttl_1d;
srcache_store_statuses 200;
proxy_pass http://192.168.0.61:900;
srcache_store_skip $no_store;

…
}

Find the debug logs when firing following urls

curl -i http://192.168.0.160/index.html < served from the memcached
curl -i http://192.168.0.160/santa.jpg < not served from the memcached
but
the request hits the memcached

error.log
http://pastebin.com/jbMm83qh

[root@localhost conf.d]# nginx -V
nginx version: nginx/1.4.1
built by gcc 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
–lock-path=/var/run/nginx.lock
–http-client-body-temp-path=/var/cache/nginx/client_temp
–http-proxy-temp-path=/var/cache/nginx/proxy_temp
–http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
–http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
–http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
–group=nginx
–with-http_ssl_module --with-http_realip_module
–with-http_addition_module
–with-http_sub_module --with-http_dav_module --with-http_gunzip_module
–with-http_gzip_static_module --with-http_random_index_module
–with-http_secure_link_module --with-http_stub_status_module
–with-mail_ssl_module --with-file-aio --without-mail_smtp_module
–without-mail_imap_module --without-mail_pop3_module --with-debug
–with-http_spdy_module
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/ngx_devel_kit-master
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/echo-nginx-module-master
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/set-misc-nginx-module-master
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/srcache-nginx-module-master
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/nginx-sticky-module-1.1
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/nginx_upstream_check_module-master
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/memc-nginx-module-master
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.4.1/contrib/nginx_cross_origin_module-master
–with-cc-opt=‘-O2 -g’

Really appreciate your support here.

Posted at Nginx Forum:

n1xman · June 18, 2013, 5:37pm

Hi agentzh,

Thanks for the response. It is not happening on my uat site as I have
redone
the configuration and tested with the full setup. However it happened on
my
try out setup where I did not want image/* content-type to be cached.

2013/06/12 15:51:43 [info] 11016#0: *19 key: “%252fsanta.jpg” was not
found
by memcached while reading response header from upstream, client:
192.168.0.61, server: -, request: “GET /santa.jpg HTTP/1.1”, subrequest:
“/memc”

Maybe my configuration problem

Thanks again for the support and the setup is now cached by MIME types
also;
we are running ngx_srcache without a problem.

Posted at Nginx Forum:

n1xman · June 13, 2013, 9:39pm

Hello!

On Wed, Jun 12, 2013 at 3:51 AM, n1xman wrote:

However, I have noticed every srcache_fetch GET /memc subrequest hits the
memcached even though srcache_store skipped content-type which is not
defined.

The cache fetch operation must be performed before sending a request
to the backend while the Content-Type response header is only
available after the backend request is already sent. You’re having a
chicken and egg problem here. And I don’t see how the srcache_fetch
can be skipped without a Content-Type response header available for
checks.

This will keep busy the memcached server and I think srcache_fetch GET
subrequest should not executed if srcache_store_skip triggered first.

Are you sure?

srcache_store_skip always runs after srcache_fetch for a particular
request. The caching workflow looks like this:

srcache_fetch
send backend request if it's a cache miss
receive backend response (headers)
srcache_store

I cannot see how srcache_store_skip can be triggered before
srcache_fetch without sending a backend request to get the
Content-Type response header.

Best regards,
-agentzh