Nginx crashing every few hours?

Server NGINX
1

Hi,
last few years I using nginx on very busy websites and all time all
working
fine, never I don`t have problem with nginx crashing but now I move my
websites to new server and I using same settings like on old server but
now
nginx crashing every few hours, here is nginx -V info:
nginx version: nginx/1.8.0
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC)
configure arguments: --with-http_stub_status_module
–without-mail_pop3_module --without-mail_imap_module
–without-mail_smtp_module --without-http_autoindex_module
–without-http_browser_module --without-http_geo_module
–without-http_empty_gif_module --without-http_memcached_module
–without-http_ssi_module --with-debug

centos-release-6-7.el6.centos.12.3.x86_64

And here is log where nginx stop to work:
2015/12/04 08:09:45 [debug] 100691#0: *780601 post rewrite phase: 3
2015/12/04 08:09:45 [debug] 100691#0: *780601 generic phase: 4
2015/12/04 08:09:45 [debug] 100691#0: *780601 generic phase: 5
2015/12/04 08:09:45 [debug] 100691#0: *780601 access phase: 6
2015/12/04 08:09:45 [debug] 100691#0: *780601 access phase: 7
2015/12/04 08:09:45 [debug] 100691#0: *780601 post access phase: 8
2015/12/04 08:09:45 [debug] 100691#0: *780601 content phase: 9
2015/12/04 08:09:45 [debug] 100691#0: *780601 content phase: 10
2015/12/04 08:09:45 [debug] 100691#0: *780601 http filename:
“/home/mmmm/public_html/logo.gif”
2015/12/04 08:09:45 [debug] 100691#0: *780601 add cleanup:
0000000000A433F8
2015/12/04 08:09:45 [debug] 100691#0: *780601 cached open file:
/home/mmmm/public_html/logo.gif, fd:49, c:1, e:0, u:50
2015/12/04 08:09:45 [debug] 100691#0: *780601 http static fd: 49
2015/12/04 08:09:45 [debug] 100691#0: *780601 http set discard body
2015/12/04 08:09:45 [debug] 100691#0: *780601 HTTP/1.1 200 OK
2015/12/04 08:09:45 [debug] 100691#0: *780601 write new buf t:1 f:0
0000000000A435D8, pos 0000000000A435D8, size: 234 file: 0, size: 0
2015/12/04 08:09:45 [debug] 100691#0: *780601 http write filter: l:0 f:0
s:234
2015/12/04 08:09:45 [debug] 100691#0: *780601 http output filter
“/logo.gif?”
2015/12/04 08:09:45 [debug] 100691#0: *780601 http copy filter:
“/logo.gif?”
2015/12/04 08:09:45 [debug] 100691#0: *780601 malloc:
0000000000A390D0:5126
2015/12/04 08:09:45 [debug] 100691#0: *780601 read: 49,
0000000000A390D0,
5126, 0
2015/12/04 08:09:45 [debug] 100691#0: *780601 write old buf t:1 f:0
0000000000A435D8, pos 0000000000A435D8, size: 234 file: 0, size: 0
2015/12/04 08:09:45 [debug] 100691#0: *780601 write new buf t:1 f:0
0000000000A390D0, pos 0000000000A390D0, size: 5126 file: 0, size: 0
2015/12/04 08:09:45 [debug] 100691#0: *780601 http write filter: l:1 f:0
s:5360
2015/12/04 08:09:45 [debug] 100691#0: *780601 http write filter limit 0
2015/12/04 08:09:45 [debug] 100691#0: *780601 writev: 5360 of 5360
2015/12/04 08:09:45 [debug] 100691#0: *780601 http write filter
0000000000000000
2015/12/04 08:09:45 [debug] 100691#0: *780601 http copy filter: 0
“/logo.gif?”
2015/12/04 08:09:45 [debug] 100691#0: *780601 http finalize request: 0,
“/logo.gif?” a:1, c:1
2015/12/04 08:09:45 [debug] 100691#0: *780601 set http keepalive handler
2015/12/04 08:09:45 [debug] 100691#0: *780601 http close request
2015/12/04 08:09:45 [debug] 100691#0: *780601 http log handler
2015/12/04 08:09:45 [debug] 100691#0: *780601 run cleanup:
0000000000A433F8
2015/12/04 08:09:45 [debug] 100691#0: *780601 close cached open file:
/home/mmmm/public_html/logo.gif, fd:49, c:0, u:50, 0
2015/12/04 08:09:45 [debug] 100691#0: *780601 free: 0000000000A390D0
2015/12/04 08:09:45 [debug] 100691#0: *780601 free: 0000000000A42180,
unused: 5
2015/12/04 08:09:45 [debug] 100691#0: *780601 free: 0000000000A43190,
unused: 2480
2015/12/04 08:09:45 [debug] 100691#0: *780601 free: 000000000097C060
2015/12/04 08:09:45 [debug] 100691#0: *780601 hc free: 0000000000000000
0
2015/12/04 08:09:45 [debug] 100691#0: *780601 hc busy: 0000000000000000
0
2015/12/04 08:09:45 [debug] 100691#0: *780601 tcp_nodelay
2015/12/04 08:09:45 [debug] 100691#0: *780601 reusable connection: 1
2015/12/04 08:09:45 [debug] 100691#0: *780601 event timer add: 86:
4000:1449212989792
2015/12/04 08:09:45 [debug] 100691#0: *780601 post event
00007F6CAB710EC8
2015/12/04 08:09:45 [debug] 100691#0: posted event 00007F6CAB710EC8
2015/12/04 08:09:45 [debug] 100691#0: *780601 delete posted event
00007F6CAB710EC8
2015/12/04 08:09:45 [debug] 100691#0: *780601 http keepalive handler
2015/12/04 08:09:45 [debug] 100691#0: *780601 malloc:
000000000097C060:1024
2015/12/04 08:09:45 [debug] 100691#0: *780601 recv: fd:86 -1 of 1024
2015/12/04 08:09:45 [debug] 100691#0: *780601 recv() not ready (11:
Resource
temporarily unavailable)
2015/12/04 08:09:45 [debug] 100691#0: *780601 free: 000000000097C060
2015/12/04 08:09:45 [debug] 100691#0: worker cycle
2015/12/04 08:09:45 [debug] 100691#0: accept mutex locked
2015/12/04 08:09:45 [debug] 100691#0: epoll timer: 164
2015/12/04 08:09:45 [notice] 100692#0: signal 15 (SIGTERM) received,
exiting
2015/12/04 08:09:45 [info] 100692#0: epoll_wait() failed (4: Interrupted
system call)
2015/12/04 08:09:45 [debug] 100692#0: timer delta: 65
2015/12/04 08:09:45 [notice] 100692#0: exiting
2015/12/04 08:09:45 [debug] 100692#0: flush files
2015/12/04 08:09:45 [debug] 100692#0: run cleanup: 00000000009F8A80
2015/12/04 08:09:45 [debug] 100692#0: cleanup resolver
2015/12/04 08:09:45 [debug] 100692#0: run cleanup: 000000000098A138
2015/12/04 08:09:45 [debug] 100692#0: open file cache cleanup
2015/12/04 08:09:45 [notice] 100691#0: signal 15 (SIGTERM) received,
exiting
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/mmmm/public_html/wp-content/themes/u-design/scripts/admin/uploadify/uploadify.css
2015/12/04 08:09:45 [notice] 100690#0: signal 15 (SIGTERM) received,
exiting
2015/12/04 08:09:45 [info] 100691#0: epoll_wait() failed (4: Interrupted
system call)
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/mmmm/public_html/piwik.php
2015/12/04 08:09:45 [debug] 100691#0: timer delta: 18
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/mmmm/public_html/piwik.php, fd:-1, c:0, u:1, 1
2015/12/04 08:09:45 [notice] 100691#0: exiting
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/sharedt/public_html/some.php
2015/12/04 08:09:45 [debug] 100691#0: flush files
2015/12/04 08:09:45 [debug] 100690#0: wake up, sigio 0
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/sharedt/public_html/some.php, fd:161, c:0, u:11, 1
2015/12/04 08:09:45 [debug] 100691#0: run cleanup: 00000000009F8A80
2015/12/04 08:09:45 [debug] 100690#0: child: 0 100691 e:0 t:0 d:0 r:1
j:0
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/turboi/public_html/some.php
2015/12/04 08:09:45 [debug] 100691#0: cleanup resolver
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/turboi/public_html/some.php, fd:127, c:0, u:4, 1
2015/12/04 08:09:45 [debug] 100690#0: child: 1 100692 e:0 t:0 d:0 r:1
j:0
2015/12/04 08:09:45 [debug] 100691#0: run cleanup: 000000000098A138
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/turboi/public_html/i/some.php
2015/12/04 08:09:45 [debug] 100690#0: termination cycle: 50
2015/12/04 08:09:45 [debug] 100691#0: open file cache cleanup
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/turboi/public_html/i/some.php, fd:-1, c:0, u:1, 1
2015/12/04 08:09:45 [debug] 100691#0: delete cached open file:
/home/mmmm/public_html/some.php
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/turboi/public_html/some.php
2015/12/04 08:09:45 [debug] 100691#0: close cached open file:
/home/mmmm/public_html/some.php, fd:140, c:0, u:4, 1
2015/12/04 08:09:45 [debug] 100690#0: sigsuspend
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/turboi/public_html/some.php, fd:135, c:0, u:214, 1
2015/12/04 08:09:45 [debug] 100691#0: delete cached open file:
/home/mmmm/public_html/images/vid.png
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/turboi/public_html/some.php
2015/12/04 08:09:45 [debug] 100691#0: close cached open file:
/home/mmmm/public_html/images/vid.png, fd:119, c:0, u:2, 1
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/turboi/public_html/some.php, fd:79, c:0, u:223, 1
2015/12/04 08:09:45 [debug] 100691#0: delete cached open file:
/home/mmmm/public_html/images/default.png
2015/12/04 08:09:45 [debug] 100691#0: close cached open file:
/home/mmmm/public_html/images/default.png, fd:-1, c:0, u:1, 1
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/mmmm/public_html/some.php
2015/12/04 08:09:45 [debug] 100691#0: delete cached open file:
/home/mmmm/public_html/piwik.php
2015/12/04 08:09:45 [debug] 100692#0: close cached open file:
/home/mmmm/public_html/some.php, fd:55, c:0, u:111, 1
2015/12/04 08:09:45 [debug] 100691#0: close cached open file:
/home/mmmm/public_html/piwik.php, fd:-1, c:0, u:1, 1
2015/12/04 08:09:45 [debug] 100692#0: delete cached open file:
/home/mmmm/public_html/search.php
2015/12/04 08:09:45 [debug] 100691#0: delete cached open file:
/home/mmmm/public_html/wp-admin/admin-ajax.php

Where can be problem?

Thanks.

Posted at Nginx Forum:

2

Hello!

On Fri, Dec 04, 2015 at 07:57:42AM -0500, x-man wrote:

last few years I using nginx on very busy websites and all time all working
fine, never I don`t have problem with nginx crashing but now I move my
websites to new server and I using same settings like on old server but now
nginx crashing every few hours, here is nginx -V info:

[…]

2015/12/04 08:09:45 [notice] 100692#0: signal 15 (SIGTERM) received, exiting
…
2015/12/04 08:09:45 [notice] 100692#0: exiting
…
2015/12/04 08:09:45 [notice] 100691#0: signal 15 (SIGTERM) received, exiting
…
2015/12/04 08:09:45 [notice] 100690#0: signal 15 (SIGTERM) received, exiting
…
2015/12/04 08:09:45 [notice] 100691#0: exiting
…
2015/12/04 08:09:45 [debug] 100690#0: wake up, sigio 0
…
2015/12/04 08:09:45 [debug] 100690#0: child: 0 100691 e:0 t:0 d:0 r:1 j:0
…
2015/12/04 08:09:45 [debug] 100690#0: child: 1 100692 e:0 t:0 d:0 r:1 j:0

[…]

Where can be problem?

As per logs, someone or something send SIGTERM to all nginx
processes, including master (pid 100690) and all workers (100691,
100692). Check your server to find out how this happened.
Usually such things happen due to bugs in log rotation scripts or
something similar.

–
Maxim D.
http://nginx.org/

3

Hello,

It can also be oomkiller killing nginx due to low resources, you should
monitor & check the other logs.

2015-12-04 16:43 GMT+02:00 Maxim D. [email protected]:

4

egrep -i -r ‘killed process’ /var/log/

result nothing…

Posted at Nginx Forum:

5

I also thinking about it but I can`t find what doing it…on this server
I
have cpanel and csf firewall with processes monitoring but csf sending
e-mail when do something…

Posted at Nginx Forum: