SSLv2 bad detection, patch

Problem:
old web application supporting SSLv2 only can’t connect to the
web service migrated behind ssl-offloading nginx.

Analysis:
nginx considers connection as plain http. Traffic analysis using
various client options shows that 0x80 isn’t the only possibility
to show up as first byte in ssl client hello.

Nginx code excerpt ( nginx-0.7.67, src/http/ngx_http_request.c:551 ):
if (buf[0] == 0x80 /* SSLv2 / || buf[0] == 0x16 / SSLv3/TLSv1
*/) {
^^

openssl code excerpt ( openssl-0.9.8k, as such version is installed on
server side, ssl/s23_srvr.c:268 ):

            if ((p[0] & 0x80) && (p[2] == SSL2_MT_CLIENT_HELLO))
                      ^^

Difference in SSL logic detection underlined.

Solution:
Fix SSLv2 detection. Patch attached.

Hello!

On Thu, Jul 01, 2010 at 05:00:49PM +0200, Mirosław Jaworski wrote:

Problem:
old web application supporting SSLv2 only can’t connect to the
web service migrated behind ssl-offloading nginx.

Analysis:
nginx considers connection as plain http. Traffic analysis using
various client options shows that 0x80 isn’t the only possibility
to show up as first byte in ssl client hello.

A bit more details analisys:

Non-0x80 first byte indicate that message length is more than 255
bytes. Session id and challenge data in client hello are
limited to 16 and 32 bytes, and fixed fields use 9 bytes, this
gives us something about 198 bytes for cipher specs, i.e. 66
ciphers listed. Typical openssl have about 30 ciphers by default,
so 66 isn’t something unrealistic.

In fact, first byte may not even contain high bit set if message
length is more than 32k. But it’s really unlike to happen in real
life (and won’t work with openssl at least as far as I see).

See here for SSLv2 specs:

http://www.mozilla.org/projects/security/pki/nss/ssl/draft02.html

Difference in SSL logic detection underlined.
Actually, the openssl also checks first byte of message as you can
see here. But just checking for & 0x80 should be fine as it
catches typical http-to-https case.

Solution:
Fix SSLv2 detection. Patch attached.

Thanks for the patch. Looks good for me.

Maxim D.