Recently I tried setting up a basic nginx reverse proxy in production on
Ubuntu 14.04 using their default supported version of nginx 1.4.6.
Basic config as follows:
server {
listen 127.0.0.1:443;
server_name myhost.ca;
ssl on;
ssl_certificate /etc/nginx/certs/cert.chained.with.intermediates.crt
ssl_certificate_key /etc/nginx/certs/cert.key
ssl_dhparam /etc/nginx/certs/dhparams.pem;
ssl_session_timeout 5m;
ssl_session_cache shared:test_cache:5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers
'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:
ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:
!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA’;
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/proxy.access.log;
error_log /var/log/nginx/proxy.error.log;
location / {
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://10.1.10.1;
}
}
Config worked pretty good in testing, but when we put it in production,
we
quickly started seeing intermittent handshake failures.
The handshakes were being rejected by the server with errors like this
in
the error log:
2016/02/16 13:30:18 [info] 6470#0: *6349 SSL_do_handshake() failed (SSL:
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad
record
mac) while SSL handshaking, client: x.x.x.x, server: x.x.x.x:443
And sometimes from the client like this:
2016/02/16 13:27:51 [info] 6468#0: *6226 peer closed connection in SSL
handshake while SSL handshaking, client: x.x.x.x, server: x.x.x.x:443
Upon further investigation we discovered that the same clients were more
or
less “randomly” effected by this handshake error on an intermittent
basis,
so one request might work but then the next would fail.
Initially we didn’t have any ssl_session_cache enabled, but adding that
shared cache setting above had no effect on the random handshake errors.
Thought it might be an openssl issue, so we tried updating from ubuntu’s
default version of 1.0.1f to 1.0.2f from source, but that had no impact
on
the clients receiving the handshake error.
Subsequently, we switched the reverse proxy on the same system, with the
same configuration (i.e. supported protocols and ciphers) from nginx to
apache, and the intermittent handshake errors have gone away.
I’d still like to know what was wrong with our nginx setup to be causing
this in the first place, anyone have any ideas?