Hi
I have recently upgraded our fleet of Ubuntu servers to nginx 1.8.0
via the nginx/stable PPA1. After the upgrade we are seeing two -
probably related - problems. They are:
-
When the log files are rotated by logrotate(8) nginx doesn’t start
writing to the new log files until it’s reloaded, restarted or has
been “/etc/init.d/nginx rotate”'ed. -
The ulimit setting for number of open files specified inside
/etc/default/nginx no longer takes effect.
The first bug I debugged being down to the "invoke-rc.d nginx rotate
/dev/null 2>&1" command being ran after the log is rotated. I tried
running the command manually and the output doesn’t look very
promising:
$ sudo invoke-rc.d nginx rotate
initctl: invalid command: rotate
Try `initctl --help' for more information.
invoke-rc.d: initscript nginx, action "rotate" failed.
I worked around that issue with replacing “invoke-rc.d” inside
/etc/logrotate.d/nginx with “service”, and since then the logs has
been rotated correctly.
When debugging the second problem, I found out that the 1.8.0 version
of the nginx-common package - besides the init.d script - also
includes an Upstart configuration. I don’t know why both types of
service scripts are included in the configuration, because it
introduces a couple of problems:
- On boot up it’s a race between if nginx is started by Upstart or by
the init.d script - Upstart seems to win most of the times from what I
have seen. - The Upstart configuration doesn’t support the “rotate” call, which
is causing invoke-rc.d to fail in the log rotation configuration,
since it prefers to call an Upstart service rather than an init.d
script. - /etc/default/nginx is no longer used by the Upstart configuration,
possibly causing big problems for people (like me) who had to increase
the limit of number of files descriptors nginx is allowed to have
open.
As a test I started up an EC2 machine based on an image which has the
nginx-common 1.6.3-1+trusty0 package installed. The machine only has
an init.d script for nginx and thus nginx is started through that.
I then upgraded nginx on the machine and got it to nginx-common
1.8.0-1+trusty1. After the upgrade nginx was still started through the
init.d script, and thus any changes in /etc/default/nginx would have
been applied. Log rotation will however not work at this stage, since
the Upstart script means invoke-rc.d in the log rotation configuration
will fail to send a USR1 signal to nginx.
The frightening thing is then, when the machine is rebooted and nginx
the next time is started by Upstart, the changes make to
/etc/default/nginx no longer applies, but it can be difficult to spot
as nginx otherwise start up normally.
I think this is a pretty bad problem and something that should be
fixed sooner rather than later. I don’t really care if we go for
Upstart or init.d, as long as there’s only one way to do it and there
aren’t files included with the package that may lead you to think it
works in another way.
–
Jeppe Toustrup