Nginx and http/2

I could use some help with this one - I took a big leap with enabling
http/2 support and I got knocked back really quick. There seems to be an
issue with POSTs and it seems to be more pronounced with ios devices (as
much as you can trust user agents) but there were some non-ios devices
that
seemed to be having issues also. Unfortunately I had to pull the changes
quickly so I didn’t get to capture too much debugging information (plus
all
the connections were via tls 1.2 w/ diffie-hellman so even if I had
quickly
taken a packet dump I wouldn’t have been able to decrypt it).

So I built a version of curl with http/2 support to try and reproduce:

curl 7.49.1 (x86_64-pc-linux-gnu) libcurl/7.49.1 OpenSSL/1.0.2h
nghttp2/1.11.1
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s
rtsp
smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL TLS-SRP HTTP2 UnixSockets

I did a test POST request with http/1 and I got a 200 response.

I did the same POST request via http/2 and the tls handshake completed
but
the connection was closed shortly thereafter. The error I see in the log
file is “client sent stream with data before settings were acknowledged
while processing HTTP/2 connection” I see other references to this error
on
POST requests when I googled for it, but I didn’t see a solution. This
sounds like an interoperability issue but I’d be shocked if I’m the
first
one to find something like that.

This is also different then the errors I was seeing earlier in the week

in those cases it looked like nginx as receiving the POST requests via
http/2, and forwarding to an HAProxy upstream via http/1.1. From there
the
requests were dispatched to servers, also via http/1.1. One set of
backend
servers seemed to be getting duplicate requests (but only of requests
that
originated as HTTP/2 POSTs) and the other set running a java service
seemed
to dislike the content received and they were closing the connections.
I’m
still trying to reproduce those transactions in a dev environment w/
diffie-hellman disabled so I can get a packet capture and get a better
idea
of what is happening.

On Saturday 25 June 2016 21:00:37 CJ Ess wrote:

file is “client sent stream with data before settings were acknowledged
while processing HTTP/2 connection” I see other references to this error on
POST requests when I googled for it, but I didn’t see a solution. This
sounds like an interoperability issue but I’d be shocked if I’m the first
one to find something like that.

[…]

You’re not the first one, so the problem was resolved by this change:
http://hg.nginx.org/nginx/rev/ce94f07d5082 (nginx 1.11.0)

wbr, Valentin V. Bartenev

Thank you very much for the pointer to the change, I’m going give that a
shot ASAP.

On Sun, Jun 26, 2016 at 8:55 AM, Valentin V. Bartenev [email protected]

I finally had a chance to test this, I applied ce94f07d5082 to the
1.9.15
code – it applied cleanly and compiled cleanly. However, my test post
request over http2 with curl failed with the same “client sent stream
with
data before settings were acknowledged while processing HTTP/2
connection”
error. Are there other changes in 1.11.x that I might need? Do I need to
explicitly enable the preread functionality? It looked to me as-if it
was
enabled by default.

The error does seem to be 100% reproducible, the command I am running
is:

curl --http2 -k -v -X POST -H “Content-Type: application/json” -d “{}” "
https://test-server_name/"

And my curl is: curl 7.49.1 (x86_64-pc-linux-gnu) libcurl/7.49.1
OpenSSL/1.0.2h nghttp2/1.11.1

On Sun, Jun 26, 2016 at 8:55 AM, Valentin V. Bartenev [email protected]

On Monday 27 June 2016 17:33:12 CJ Ess wrote:

I finally had a chance to test this, I applied ce94f07d5082 to the 1.9.15
code – it applied cleanly and compiled cleanly. However, my test post
request over http2 with curl failed with the same “client sent stream with
data before settings were acknowledged while processing HTTP/2 connection”
error. Are there other changes in 1.11.x that I might need? Do I need to
explicitly enable the preread functionality? It looked to me as-if it was
enabled by default.

The answer to both questions no, you don’t need. But it looks like
you’re
still running the old binary.

The error does seem to be 100% reproducible, the command I am running is:

curl --http2 -k -v -X POST -H “Content-Type: application/json” -d “{}” "
https://test-server_name/"

And my curl is: curl 7.49.1 (x86_64-pc-linux-gnu) libcurl/7.49.1
OpenSSL/1.0.2h nghttp2/1.11.1

[…]

Why don’t you just install nginx 1.11?

wbr, Valentin V. Bartenev

You were correct, there was a typeo in my rpm spec that kept the diff
from
applying but didn’t kill the build. The curl request is working now! Now
I
need to see if those other POST requests are working.

I’m trying to use http/2 with Nginx 1.9.15 from Openresty - I don’t know
if
they modify the core or if I can just drop in 1.11.0 as a replacement.

I’ll see if I can put a misspelling in that error message so I can be
sure
I’m running the new binary.

On Mon, Jun 27, 2016 at 5:45 PM, Valentin V. Bartenev [email protected]