Nginx + cookies + firefox = 400 Bad Request

Hi guys, this week is was about to move partially to nginx for all
static content. The full switch was planned in about a month on a new
server. I followed this guide
http://www.dikant.de/2008/07/10/nginx-as-a-reverse-proxy-for-apache/ and
it worked instantly. After some minutes I got some t-calls about saying
that the page isnt reachable with a 400 - Bad Request for them. It runs
a vbulletin-forum and a Mantis-Bugtracker, it happens on both.

Some hours later I could nail that problem down to nginx (same with 0.5x
from etch and 0.632 from lenny) and firefox users (versions 2&3). It
works again if the related domain-cookies are deleted. apache2 doesnt
cause any trouble, same for IE/Opera-users.

Tried a quick google search and I found endless posts on a lot of sites
with the same 400 problem and ff after they switched to nginx (even
famous ones like electronicarts). Sadly none if these contain a solution
besides cleaning the cookies but that doesnt seem to help for long
according to that posts.

As I have over 60% FF-users Im a little afraid of the results and
switched back to pure apache for the moment. Is there any known solution
to this problem on serverside? May a selfcompiled-0.7-version help on
this? Couldnt find anything related in the changelog.

Many thanks in advance
Oliver

It might be but I don’t have that problem with my moderately busy
vBulletin
board and 31% of my visitors (including myself) use Firefox. The
difference
is that I use php-fcgi not Apache. Looking at Google search the problem
appears with other apps but the common denominator in most cases seems
to be
when nginx is being used as a reverse proxy and only with Firefox. The
problem is more likely with Firefox than with vBulletin. As for number
of
cookies, my browser has 11 from my vBulletin installation. The average
user
has one fewer since they don’t have an admin control panel cookie. In
contrast, I have 28 cookies from CNN.com (which uses nginx to serve some
of
the content that I browse), 30 from VerizonWireless (my cell phone
company),
and 16 from American Express, so 11 (or 10 as a typical user might have)
does not seem “enormous”.

Consider a brief test proxying your php requests to php-cgi (I use
php-fpm
but you can use spawn-fcgi from lighttpd and you probably won’t need to
recompile php for this test). There are lots of “recipes” online for
doing
so. If the problem resolves then you can make a more informed decision.
It
appears that the problem has existed in at least some versions of nginx
0.7.x - see

so a recompile may not help. I don’t know if it’s been rectified at that
site but I have visited it without issue. They are still using nginx but
how
it is configured I do not know. If you decide to do this, also consider
running some benchmarks using a simple php script: Apache alone, nginx
as
reverse proxy to Apache, nginx using fastcgi.

Good luck!

Jim

Hello!

On Sun, Dec 14, 2008 at 01:26:59PM +0100, Oliver Pestring wrote:

works again if the related domain-cookies are deleted. apache2 doesnt
to this problem on serverside? May a selfcompiled-0.7-version help on
this? Couldnt find anything related in the changelog.

You should tune large_client_header_buffers in your nginx config, see
http://wiki.codemongers.com/NginxHttpCoreModule#large_client_header_buffers
for details.

The problem AFAIK is vbulletin which sets enormous number of
various cookies.

Maxim D.

That would explain why it doesn’t happen on my installation as I use
database to track thread/forum read info for users, not cookie. That’s a
quick setting change in the vBulletin admin panel to fix though it does
put
a bit more load on the processor.

Hello!

On Sun, Dec 14, 2008 at 09:17:02PM -0500, Jim O. wrote:

It might be but I don’t have that problem with my moderately busy vBulletin
board and 31% of my visitors (including myself) use Firefox. The difference
is that I use php-fcgi not Apache. Looking at Google search the problem
appears with other apps but the common denominator in most cases seems to be
when nginx is being used as a reverse proxy and only with Firefox. The
problem is more likely with Firefox than with vBulletin. As for number of

Other browsers seems to have lower limits on total cookie size, and
hence the problem doesn’t manifest itself (instead, some cookies
probably just discarded by other browsers).

cookies, my browser has 11 from my vBulletin installation. The average user
has one fewer since they don’t have an admin control panel cookie. In
contrast, I have 28 cookies from CNN.com (which uses nginx to serve some of
the content that I browse), 30 from VerizonWireless (my cell phone company),
and 16 from American Express, so 11 (or 10 as a typical user might have)
does not seem “enormous”.

Yep, I was wrong in my assumption (just did a quick-review of
vbulletin code). It doesn’t create many cookies, it just uses
several ones that may grow really big.

Anyway, the problem is that client sent too long Cookie header to
nginx.

Quick fix is to enlarge large_client_header_buffers. And the next
thing to tune is proxy_buffer_size, which in turn will require
enlargement.

More correct fix would be to limit backend somehow, but it’s out
of scope of this list.

Consider a brief test proxying your php requests to php-cgi (I use php-fpm
but you can use spawn-fcgi from lighttpd and you probably won’t need to
recompile php for this test). There are lots of “recipes” online for doing
so. If the problem resolves then you can make a more informed decision. It
appears that the problem has existed in at least some versions of nginx
0.7.x - see http://www.linuxquestions.org/questions/project.php?issueid=30
so a recompile may not help. I don’t know if it’s been rectified at that

BTW, looks like vbulletin on www.linuxquestions.org was patched to
avoid bbthread_lastview cookie (the one that grows for each forum
tread visited by user).

Maxim D.

Maxim D. wrote:

You should tune large_client_header_buffers in your nginx config, see
http://wiki.codemongers.com/NginxHttpCoreModule#large_client_header_buffers
for details.

The problem AFAIK is vbulletin which sets enormous number of
various cookies.

Many thanks guys! It seems that the issue doesnt occur with the
0.7.26-version. I compiled it manually yesterday evening and vbulletin
and Mantis work perfectly with the previous config until now.
Just to be sure Ive now followed your advices and added some larger
values for large_client_header_buffers and proxy_buffer_size too. Also
switched from cookie-based activity to db-based for vb.

My testsystem is still on lennys 0.6.32 and here an increased
large_client_header_buffers was even enough to get it working again.
Perfect

On Mon, Dec 15, 2008 at 05:59:07PM -0800, Neil S. wrote:

I’m continuing to also see issues with nginx returning 400s, I posted
another thread about this earlier. The suggestions here are worth
trying, hopefully they help!

For the large_client_header_buffers setting, what values are recommended?

To see 400’ reasons you should set error_log level to info:

error_log /path/to/log info;

They may be not associated with bufers at all.

As to large_client_header_buffers, Apache’s default one header limit is
8K:
http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestfieldsize

nginx’s defaults are

client_header_buffer_size 1k;
large_client_header_buffers 4 4k;
# or 4 8k for sparc64/ppc64 (but not amd64)

You will see in log that these buffers are no enough, you may set

client_header_buffer_size 1k;
large_client_header_buffers 8 8k;

Setting the error_log to info, I’m seeing that a lot of the 400
requests have a corresponding entry in the error log saying “client
closed prematurely connection while reading client request line” - any
suggestions from that?

Thanks!

Tried increasing the large_client_header_buffers setting as suggested,
but it had no effect on my 400s. Any other suggestions? A bit at a
loss at this point, so open to any suggestions. Thanks!

On Fri, Dec 19, 2008 at 12:36:55AM -0800, Neil S. wrote:

Tried increasing the large_client_header_buffers setting as suggested,
but it had no effect on my 400s. Any other suggestions? A bit at a
loss at this point, so open to any suggestions. Thanks!

If causes of these 400s is “client closed prematurely connection
while reading client request line”, then you can not do anything
on nginx side. These may be user request cancels when he clicks on
a link on a page while browser is still loading images for the page.
These may be network problems.

I’m continuing to also see issues with nginx returning 400s, I posted
another thread about this earlier. The suggestions here are worth
trying, hopefully they help!

For the large_client_header_buffers setting, what values are
recommended?

Also, for proxy_buffers, I have the following:
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

Any suggestions if those should be tweaked in any manner?

Thanks!