How to make nginx loadbalancer give 404 when all upstream servers are down

Rafa_F · August 13, 2015, 11:13am

Hi folks,

I have been stuck this issue for a long time now. Searches could not
solve
my issue hence posting here.
Please help.

I am using nginx as a load balancer and ngnix.conf looks like:

worker_processes 4;

events { worker_connections 1024; }

http {
upstream ab_backend {
server :7000;
server :7000 backup;
}

server {
        listen 8000;

        location / {
                proxy_set_header X-Forwarded-For

$proxy_add_x_forwarded_for;
proxy_pass http://ab_backend;
}
}
}
IP1 and IP2 are simple httpd docker containers. From another machine I
am
running ab tool. What I am observing is: When IP1 is up the http it
responds
to request from ab. Bring IP1 and backup IP2 starts to respond to al
http
requests.

starnge thing is when both IP1, IP2 are down the nginx server itself
takes
it up and responds to the http requests from ab. Is there a way to make
nginx not behave this way?

what I am looking for is way to force http requests to be answered by
only
the upstream servers if none of them are reachable requests should get
errors instead of giving out non bona fide responses.

I am testing using ab and httperf tool and saw the bahavior in both
cases.
Even when no upstreams are up test reports no Errors or dropped
requests.
Please help

Posted at Nginx Forum:

karkunpavan · August 13, 2015, 1:12pm

On Thu, Aug 13, 2015 at 05:12:57AM -0400, karkunpavan wrote:

Hi there,

starnge thing is when both IP1, IP2 are down the nginx server itself takes
it up and responds to the http requests from ab. Is there a way to make
nginx not behave this way?

That does seem strange to me.

What output do you get from a request like

curl -i http://your-nginx:8000/

when one “upstream” is active; and when no “upstream” is active?

What does your nginx access_log say for those two requests?

f

Francis D. [email protected]

karkunpavan · August 14, 2015, 8:06am

Hi Fracis,

Thanks for taking a look. With it curl behaves as expected, but with
tools
like ab and httperf, I am not seeing bad gateway responses when both the
upstream servers are down. Read ahead for more details.

Below is info using ab request and curl request.

ab:

When one upstream server is up:

nginx console:
<<client_IP>> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45
“-”
“ApacheBench/2.3”
<<client_IP>> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45
“-”
“ApacheBench/2.3”
upstream httpd server console:
<<nginx_IP>> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45
<<nginx_IP>> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45

When both upstream servers are down:

2015/08/14 13:43:45 [error] 5#5: *14 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: “GET / HTTP/1.0”, upstream: “http://<upstream_1_IP>>:/”,
host: “<nginx_IP:”
2015/08/14 13:43:45 [error] 5#5: *14 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: “GET / HTTP/1.0”, upstream: “http://<upstream_2_IP>:/”,
host:
“<nginx_IP:”
2015/08/14 13:43:45 [error] 5#5: *14 no live upstreams while connecting
to
upstream, client: <<client_IP>>, server: , request: “GET / HTTP/1.0”,
upstream: “http://ab_backend/”, host: “<nginx_IP:”
<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502 172
“-”
“ApacheBench/2.3”
<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502 172
“-”
“ApacheBench/2.3”

as a result in bith cases ab just shows all successful responses:

ab -n 1 -c 1 http://<nginx_IP>:/

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking <<nginx_IP>> (be patient)…done
Server Software: nginx/1.9.3
Server Hostname: 10.107.53.127
Server Port: 9000
Document Path: /
Document Length: 45 bytes
Concurrency Level: 1
Time taken for tests: 0.040 seconds
Complete requests: 1
Failed requests: 0
Write errors: 0
Keep-Alive requests: 1
Total transferred: 285 bytes
HTML transferred: 45 bytes
Requests per second: 24.85 [#/sec] (mean)
Time per request: 40.234 [ms] (mean)
Time per request: 40.234 [ms] (mean, across all concurrent
requests)
Transfer rate: 6.92 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 2 2 0.0 2 2
Processing: 39 39 0.0 39 39
Waiting: 38 38 0.0 38 38
Total: 40 40 0.0 40 40

curl:

when one upstream is up:
<<client_IP>> - - [14/Aug/2015:13:58:37 +0000] “GET / HTTP/1.1” 200 45
“-”
“curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0
zlib/1.2.3 libidn/1.18 libssh2/1.2.2”

When both upstream servers are down:
2015/08/14 14:00:37 [error] 8#8: *26 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: “GET / HTTP/1.1”, upstream: “http://<upstream_1_IP>:/”,
host:
“<nginx_IP>:”
2015/08/14 14:00:37 [error] 8#8: *26 connect() failed (111: Connection
refused) while connecting to upstream, client: <<client_IP>>, server: ,
request: “GET / HTTP/1.1”, upstream: “http://<upstream_2_IP>:/”,
host:
“<nginx_IP>:”
<<client_IP>> - - [14/Aug/2015:14:00:37 +0000] “GET / HTTP/1.1” 502 172
“-”
“curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0
zlib/1.2.3 libidn/1.18 libssh2/1.2.2”

and curl gets a “502 Bad Gateway” which is right.

Any suggestions what I might be doing wrong?

Posted at Nginx Forum:

karkunpavan · August 14, 2015, 9:12am

karkunpavan Wrote:

<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502
172 “-” “ApacheBench/2.3”
<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502
172 “-” “ApacheBench/2.3”
[…]
<<client_IP>> - - [14/Aug/2015:14:00:37 +0000] “GET / HTTP/1.1” 502
172 “-” “curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2”

I see a 502 for both tools, might be an ‘ab’ thing.

Posted at Nginx Forum:

karkunpavan · August 14, 2015, 9:46am

Indeed. It’s right there I did not even see it. Thanks itpp2012 and
Francis
Daly.

since ab and httperf are behaving unexpectedly, I think best way is to
parse
the nginx logs to find out dropped requests.

Thanks a lot

Posted at Nginx Forum:

karkunpavan · August 14, 2015, 9:54am

On Fri, Aug 14, 2015 at 02:06:29AM -0400, karkunpavan wrote:

Hi there,

Thanks for taking a look. With it curl behaves as expected, but with tools
like ab and httperf, I am not seeing bad gateway responses when both the
upstream servers are down.

ab:

When one upstream server is up:

nginx console:
<<client_IP>> - - [14/Aug/2015:13:44:57 +0000] “GET / HTTP/1.0” 200 45 “-”
“ApacheBench/2.3”

Response status 200, content size 45.

When both upstream servers are down:

<<client_IP>> - - [14/Aug/2015:13:43:45 +0000] “GET / HTTP/1.0” 502 172 “-”
“ApacheBench/2.3”

Response status 502, content size 172.

as a result in bith cases ab just shows all successful responses:

ab -n 1 -c 1 http://<nginx_IP>:/

Server Port: 9000
Complete requests: 1
Failed requests: 0
HTML transferred: 45 bytes

One request, 45 bytes of content. That is the “success” case.

If the “fail” case shows the same, while the nginx logs show something
else, then your testing tool is wrong.

Any suggestions what I might be doing wrong?

ab is talking to port 9000.

Your previous config showed nginx listening on port 8000.

Might that be related?

f

Francis D. [email protected]

karkunpavan · August 14, 2015, 10:18am

nginx is listening on 8000. port 9000 is mapped to 8000 on the docker so
I
think it’s not related.

Posted at Nginx Forum: