Nginx-sticky & nginx_http_upstream_check modules not working together

Hi,

We use nginx-sticky-module for session persistence and we planned to use
nginx_http_upstream_check_module for to check the upstream service
health.

Scenario: We expect nginx_http_upstream_check_module to check the
service (return 2xx|3xx) and take it out from the upstream pool if the
“service” is not responsive. And nginx-sticky should respect the
decision taken by nginx_http_upstream and start new session to available
upstream server. However, the upstream server is up with the service
port; only the content not being served.

The problem in this scenario, Nginx continue to serve the failed
upstream server until we shutdown the upstream service port. Note that
http_upstream module status page indicates the respective service is
down upon I glitch the service.

I think nginx-sticky got the previous cookie and found the peer match
which is the failed server and continue to maintain the previous
session. If I shutdown the server/port, then sticky tries available
peers and create a new session with available upstream server.

What we need is nginx-sticky module to respect the decision taken by the
nginx_http_upstream module as the priority is “service” availability not
just the “server and port” availability. We run jboss on upstream
servers and the probability is very high the service is malfunction but
the jboss server is up. These two modules are working fine
individually.

Here is the configuration I have tried. We really need your support on
possible configuration options to get this two modules work together.

upstream cluster {
sticky;
server 192.168.0.61:8081;
server 192.168.0.61:8082;
check interval=3000 rise=2 fall=5 timeout=1000 type=http;
check_http_send “GET /keepalive.html HTTP/1.0\r\n\r\n”;
check_http_expect_alive http_2xx http_3xx;
}

server {
listen 80;
server_name localhost;

access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;

location /stat {
check_status;
}

location /test {
proxy_pass http://cluster/service1/test.html;
error_log /var/log/nginx/error.log debug;
}

Here is my nginx version:

nginx version: nginx/1.2.1-1
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
TLS SNI support disabled
configure arguments: --prefix=/etc/nginx/ --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid
–lock-path=/var/run/nginx.lock
–http-client-body-temp-path=/var/cache/nginx/client_temp
–http-proxy-temp-path=/var/cache/nginx/proxy_temp
–http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
–http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
–http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx
–group=nginx --with-http_ssl_module --with-http_realip_module
–with-http_addition_module --with-http_sub_module
–with-http_dav_module --with-http_flv_module --with-http_mp4_module
–with-http_gzip_static_module --with-http_random_index_module
–with-http_secure_link_module --with-http_stub_status_module
–with-mail_ssl_module --with-file-aio --with-debug --with-cc-opt=‘-O2
-g -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables’
–without-http_uwsgi_module --without-http_scgi_module
–without-mail_pop3_module --without-mail_imap_module
–without-mail_smtp_module
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/simpl-ngx_devel_kit-24202b4
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-set-misc-nginx-module-87d0ab2
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/mikewest-nginx-static-etags-25bfaf9
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/nginx-sticky-module-1.0
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-memc-nginx-module-8befc56
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-srcache-nginx-module-8df221e
–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/yaoweibin-nginx_upstream_check_module-dfee401

uname -rop
2.6.18-194.el5PAE i686 GNU/Linux

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)

debug logs:
service down, service port still up - http://pastebin.com/S9wJFkmR
service down, service port down- http://pastebin.com/HSjMMs75

Thanks in advance.

Posted at Nginx Forum:

2012/6/26 n1xman [email protected]

decision taken by nginx_http_upstream and start new session to available
session. If I shutdown the server/port, then sticky tries available
possible configuration options to get this two modules work together.
server {
location /test {
–conf-path=/etc/nginx/nginx.conf
–with-http_dav_module --with-http_flv_module --with-http_mp4_module

–add-module=/usr/local/hirantha/rpmbuild/BUILD/nginx-1.2.1/contrib/agentzh-echo-nginx-module-080c0a1

service down, service port down- http://pastebin.com/HSjMMs75

Thanks in advance.

Hi,

can you please open a bug report about the sticky module
(Google Code Archive - Long-term storage for Google Code Project Hosting.). I’ll look into it (no
ETA as i’m away for vacations these days).

++ Jerome

Thanks Jerome,

We will hold the nginx_http_upstream_check module until resolve this.
This module is really important for us as we use multiple Cometd
instances successfully. The only issue is sometimes the Cometd service
is not functional but the jboss port is listing.

http://code.google.com/p/nginx-sticky-module/issues/detail?id=22

Thanks in advance

Posted at Nginx Forum:

Hi,

I’m the author of the upstream check module. The upstream check only
support official round robin and ip hash module.

The sticky upstream module need an extra patch to work with this check
module. See the fair module patch as an example
(nginx_upstream_check_module/upstream_fair.patch at master · yaoweibin/nginx_upstream_check_module · GitHub).

2012/6/27 n1xman [email protected]:

Hi 姚伟斌,

Thanks for the update and it is noted. However, why we would like to use
sticky instead of ip_hash or upstream_fair as we want to make sure
persistence browser (using cookie) instead of client connecting IP as
sometimes they are behind proxies.

Posted at Nginx Forum:

Done, I have added the nginx-sticky-module.patch
(nginx_upstream_check_module/nginx-sticky-module.patch at master · yaoweibin/nginx_upstream_check_module · GitHub)
for your. You can enable it like this:

$ svn checkout http://nginx-sticky-module.googlecode.com/svn/trunk/
nginx-sticky-module
$ cd nginx-sticky-module
$ patch -p0 <
/path/to/nginx_http_upstream_check_module/nginx-sticky-module.patch
$ cd /path/to/nginx-1.0.14
$ ./configure --add-module=/path/to/nginx_http_upstream_check_module
–add-module=/path/to/nginx-sticky-module
$ make
$ make install

Note that, the nginx-sticky-module also needs the original check.patch.

See the README file for detail.

Thanks.

2012/7/20 jjk77 [email protected]:

So, does anybody did a patch of the sticky module?

@姚伟斌: Any chance you provide a patch?

best regrads
Jens

Posted at Nginx Forum:

Hi 姚伟斌,

We have tested it on test environment and it is working now; and thanks
to you! :slight_smile:

However, in our test environment we also wanted to test
nginx_tcp_proxy_module for WebSocket. We have noticed upstream server
health check and status monitor bundle to the nginx_tcp_proxy_module.

Can you tell if we compile nginx_tcp_proxy_module, we can drop the
nginx_http_upstream_check_module; and we still patch the
nginx-sticky-module with
nginx_http_upstream_check_module/nginx-sticky-module.patch and use
nginx_tcp_proxy_module for upstream health check also…?

BTW: We have tried to compile both nginx_tcp_proxy_module and
nginx_http_upstream_check_module together, following error fires during
the make process.

[hirantha@abmx-test nginx-1.2.1]$ patch -p1 <
./contrib/yaoweibin-nginx_upstream_check_module-be97c70/check_1.2.1.patch
patching file src/http/modules/ngx_http_upstream_ip_hash_module.c
patching file src/http/ngx_http_upstream_round_robin.c
patching file src/http/ngx_http_upstream_round_robin.h
[hirantha@abmx-test nginx-1.2.1]$ patch -p1 <
./contrib/yaoweibin-nginx_tcp_proxy_module-a40c99a/tcp.patch
patching file src/core/ngx_log.c
Hunk #1 succeeded at 67 (offset 1 line).
patching file src/core/ngx_log.h
Hunk #1 succeeded at 30 (offset 1 line).
patching file src/event/ngx_event_connect.h
Hunk #1 succeeded at 33 (offset 1 line).
[hirantha@abmx-test nginx-1.2.1]$ ./configure
–add-module=./contrib/yaoweibin-nginx_tcp_proxy_module-a40c99a
–add-module=./contrib/yaoweibin-nginx_upstream_check_module-be97c70
checking for OS

  • Linux 2.6.18-194.el5PAE i686
    checking for C compiler … found

    checking for openat(), fstatat() … found
    configuring additional modules
    adding module in ./contrib/yaoweibin-nginx_tcp_proxy_module-a40c99a
    checking for nginx_tcp_module … found
  • ngx_tcp_module was configured
    adding module in
    ./contrib/yaoweibin-nginx_upstream_check_module-be97c70
    checking for ngx_http_upstream_check_module … found
  • ngx_http_upstream_check_module was configured
    checking for PCRE library … found
    checking for PCRE JIT support … not found
    checking for OpenSSL library … found
    checking for zlib library … found
    creating objs/Makefile

[hirantha@abmx-test nginx-1.2.1]$make


objs/addon/modules/ngx_tcp_ssl_module.o \

objs/addon/yaoweibin-nginx_upstream_check_module-be97c70/ngx_http_upstream_check_module.o
\

objs/addon/yaoweibin-nginx_upstream_check_module-be97c70/ngx_http_upstream_check_handler.o

objs/ngx_modules.o
-lpthread -lcrypt -lpcre -lssl -lcrypto -ldl -lz
objs/addon/yaoweibin-nginx_upstream_check_module-be97c70/ngx_http_upstream_check_handler.o:(.rodata+0x40):
multiple definition of sslv3_client_hello_pkt' objs/addon/yaoweibin-nginx_tcp_proxy_module-a40c99a/ngx_tcp_upstream_check.o:(.rodata+0x0): first defined here collect2: ld returned 1 exit status make[1]: *** [objs/nginx] Error 1 make[1]: Leaving directory /usr/local/hirantha/nginx-1.2.1’
make: *** [build] Error 2

Thanks in advance.

Posted at Nginx Forum:

Hi 姚伟斌,

With the latest code, I was managed to compile both modules together
without any issue.

Thanks for the modules and great support. :slight_smile:

Posted at Nginx Forum:

Hi,

Try the latest code, you can use these modules both.

Thanks.

2012/8/9 n1xman [email protected]: