Nginx on high load (upstream prematurely closed connection while reading response header from upstre

Server NGINX
1

Dear Guys,

I am facing strange issue, during load test and peak load of more than
3k
concurrent users, get below errors in nginx logs continuously.

2016/05/18 11:23:28 [error] 15510#0: *6853 upstream prematurely closed
connection while reading response header from upstream, client: x.x.x.x,
server:abc.com, request: “GET / HTTP/1.1”, upstream:
“fastcgi://127.0.0.1:9000”, host: “10.50.x.x”

And beside of above error frequently, below error come gradually

2016/05/18 11:23:28 [error] 15510#0: *7431 recv() failed (104:
Connection
reset by peer) while reading response header from upstream, client:
x.x.x.x,
server: abc.com, request: “POST / HTTP/1.1”, upstream:
“fastcgi://127.0.0.1:9000”, host: “10.50.x.x”, referrer:
“http://10.50.x.x”

But with out peak load, no error is coming and during load besides of
above
errors continues coming site is working fine and i see 200 response code
in
access logs, but still wondering why above errors coming only in peak
load
or during load testing. Below is the require info. I already tune up the
sysctl settings for high throughput

nginx version: nginx/1.8.1
php-fpm /php version : 5.4.45

echo 0 > /proc/sys/net/ipv4/tcp_ecn
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 102400 > /proc/sys/net/ipv4/tcp_max_syn_backlog
echo 102400 > /proc/sys/net/core/somaxconn
echo 102400 > /proc/sys/net/core/netdev_max_backlog
echo 5242888 > /proc/sys/net/netfilter/nf_conntrack_max
echo 400000 > /proc/sys/net/ipv4/tcp_max_tw_buckets
echo 655300 > /proc/sys/vm/max_map_count
echo 11000 65000 > /proc/sys/net/ipv4/ip_local_port_range

Kindly please help & advise.

Posted at Nginx Forum:

2

Dear Francis D. , thankyou for your response, upstream is php-fpm
server
(127.0.0.1) and its also setup to handle huge load, during above errors,
i
see the site is opening fine. but don’t know where to catch it. PM is
also
set on demand and maximum childerns set to 90000 to handle the load, and
no
logs appears in php-fpm. Can you please suggest how to got the issue ?

Posted at Nginx Forum:

3

On Wed, May 18, 2016 at 04:50:24AM -0400, muzi wrote:

Hi there,

I am facing strange issue, during load test and peak load of more than 3k
concurrent users, get below errors in nginx logs continuously.

2016/05/18 11:23:28 [error] 15510#0: *6853 upstream prematurely closed
connection while reading response header from upstream, client: x.x.x.x,
server:abc.com, request: “GET / HTTP/1.1”, upstream:
“fastcgi://127.0.0.1:9000”, host: “10.50.x.x”

“upstream” here is your fastcgi server. The nginx message is that the
fastcgi server broke the connection.

And beside of above error frequently, below error come gradually

2016/05/18 11:23:28 [error] 15510#0: *7431 recv() failed (104: Connection
reset by peer) while reading response header from upstream, client: x.x.x.x,
server: abc.com, request: “POST / HTTP/1.1”, upstream:
“fastcgi://127.0.0.1:9000”, host: “10.50.x.x”, referrer: “http://10.50.x.x”

And that message is that the fasctcgi server broke the connection.

It is probably worth investigating what the fastcgi server thinks is
happening – perhaps it is not configured to handle the load it is
being given?

f

Francis D. [email protected]

4

On Thu, May 19, 2016 at 04:31:14AM -0400, muzi wrote:

Hi there,

upstream is php-fpm server
(127.0.0.1) and its also setup to handle huge load, during above errors, i
see the site is opening fine. but don’t know where to catch it. PM is also
set on demand and maximum childerns set to 90000 to handle the load, and no
logs appears in php-fpm. Can you please suggest how to got the issue ?

nginx says that upstream is doing something odd.

upstream is php-fpm.

I suggest checking the php-fpm docs to see how to get it to log what
it is doing – if it says nginx is doing something odd, then there is
a conflict to investigate. Otherwise, it should says something about the
connections that it is closing.

If the site is opening fine, maybe there is no problem to worry about.

f

Francis D. [email protected]

5

Thankyou once again Francis D., i am now ignoring these errors, as i
not
see any errors in nginx access logs and site is working on high load and
see
200 response code in access logs.

Posted at Nginx Forum:

6

Thankyou once again Francis D., i am now ignoring these errors, as i
not
see any errors in nginx access logs and site is working on high load and
see
200 response code in access logs.

Posted at Nginx Forum:

7

Thankyou once again Francis D., i am now ignoring these errors, as i
not
see any errors in nginx access logs and site is working on high load and
see
200 response code in access logs.

Posted at Nginx Forum: