I’m using nginx 1.10.0 and the nginx_fastcgi_cache option. I’ve noticed
that
with a high amount of requests per second (I’m not sure when it occurs
exactly, but we have ~2500RPS with about 2Gbit/s of outgoing traffic
responses) there is an issue with the fastcgi_cache files not being
freed.
lsof shows a huge amount of files as (deleted) but the space it not
being
freed. Eventually the entire partition fills up like this.
The keys_zone and max_size are both set to 500m
The fastcgi_cache_valid is set to 1m
Even when the server is idle for a while (30+ minutes) the space is not
being freed. Obviously restarting nginx manually immediately frees the
space.
Is this a bug or is there a setting I can use to perform some sort of
garbage collection?
On Sun, Jun 05, 2016 at 11:32:35AM -0400, b4a456fb-1402 wrote:
The fastcgi_cache_valid is set to 1m
Even when the server is idle for a while (30+ minutes) the space is not
being freed. Obviously restarting nginx manually immediately frees the
space.
Is this a bug or is there a setting I can use to perform some sort of
garbage collection?
The fact that a file is shown as “deleted” indicates that it was
deleted but is still open. This may happen due to legitimate
reasons - e.g., the file was deleted, but is current being served
to a client and hence it is still open. But if you see no changes
with 30+ minutes without load, there is likely a problem
somewhere. You may try debugging it further.
Some basic things to consider:
Make sure you are not using open_file_cache with some insane
parameters. If you use open_file_cache, comment it out and check
if you are still able to reproduce the problem.
Check if “nginx -V” shows any 3rd party modules. If it does,
try reproducing the problem without them.
When leaving the server idle, wait to make sure all client
connections are actually closed on socket level by clients -
that is, no connections in the ESTABLISHED state. Check lsof
when this happens.