Random Instant 502 Bad Gateway Errors

Hi,

I’m French and I’ve some problem with nginx 0.8.38 and PHP-FPM 5.3.0
with Suhosin patch :confused:
I’ve got random instant 502 Bad Gateway errors, generaly a refresh make
the page works again but it’s really anoying.

Nginx Config :

user www;
worker_processes 1;
daemon on;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

worker_rlimit_nofile 32000;

events {
    worker_connections  4096;
    use epoll;
}

http {

    include       mime.types;
    default_type  text/plain;

    limit_zone http $binary_remote_addr 1m;
    limit_zone https $binary_remote_addr 1m;

    client_body_timeout 8;
    client_header_timeout 5;
    keepalive_timeout 5;
    keepalive_requests 10;
    send_timeout 8;

    client_max_body_size 512k;

    access_log off;
    error_log off;

    client_header_buffer_size    16k;
    large_client_header_buffers  16 16k;

    output_buffers   2 32k;
    postpone_output  1460;

    sendfile         on;
    tcp_nopush       on;
    tcp_nodelay      on;
    reset_timedout_connection on;

    ignore_invalid_headers on;
    autoindex on;
    msie_padding on;

    gzip  on;
    gzip_http_version 1.1;
    gzip_vary on;
    gzip_comp_level 6;
    gzip_proxied any;
    gzip_types text/plain text/css application/json 
application/x-javascript text/xml application/xml application/xml+rss 
text/javascript;
    gzip_buffers 16 16k;
    gzip_disable “MSIE [1-6].(?!.*SV1)”;

    server
    {
      listen 80 default rcvbuf=8192 sndbuf=16384 backlog=65536;
      index  index.html index.htm index.php;
      root   /home/www/;

      limit_conn http 10;

      access_log off;

     location ~* \.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ {
      expires 3600;
      break;
     }

     location ~ \.php(/|$) {
         include fastcgi_params;
     }

     location ~ \.php5(/|$) {
         include fastcgi_params;
     }

     location ~ \.php4(/|$) {
         include fastcgi_params;
     }

     location /cache/status {
         stub_status on;
         access_log off;
     }
    }

Fastcgi params :

fastcgi_connect_timeout 5;
fastcgi_send_timeout 8;
fastcgi_buffer_size 512k;
fastcgi_buffers 64 64k;
fastcgi_busy_buffers_size 512k;
fastcgi_temp_file_write_size 512k;
fastcgi_ignore_client_abort on;
fastcgi_intercept_errors on;
fastcgi_read_timeout 45;

fastcgi_pass   unix:/tmp/php.socket;
fastcgi_index  index.php;
fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;
fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  GATEWAY_INTERFACE  CGI;
fastcgi_param  SERVER_SOFTWARE    RD;
fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;
fastcgi_param  REDIRECT_STATUS    200;

PHP-FPM conf :

<?xml version="1.0" ?>
<configuration>
  <section name="global_options">
    <value name="pid_file">/var/run/php5-fpm.pid</value>
    <value name="error_log">/var/log/php-fpm.log</value>
    <value name="log_level">warn</value>
    <value name="emergency_restart_threshold">60</value>
    <value name="emergency_restart_interval">1m</value>
    <value name="process_control_timeout">4s</value>
    <value name="daemonize">yes</value>
  </section>

  <workers>
    <section name="pool">
      <value name="name">RD</value>
      <value name="listen_address">/tmp/php.socket</value>
      <value name="listen_options">
        <value name="backlog">-1</value>
        <value name="owner">www</value>
        <value name="group">www</value>
        <value name="mode">0666</value>
      </value>

      <value name="php_defines">
      </value>

      <value name="user">www</value>
      <value name="group">www</value>

      <value name="pm">
        <value name="style">static</value>
        <value name="max_children">150</value>
        <value name="apache_like">
          <value name="StartServers">25</value>
          <value name="MinSpareServers">5</value>
          <value name="MaxSpareServers">35</value>
        </value>
      </value>

      <value name="request_terminate_timeout">0s</value>
      <value name="request_slowlog_timeout">0s</value>
      <value name="slowlog">/var/log/php-fpm.log.slow</value>
      <value name="rlimit_files">1024</value>
      <value name="rlimit_core">0</value>
      <value name="chroot"></value>
      <value name="chdir"></value>
      <value name="catch_workers_output">yes</value>
      <value name="max_requests">65536</value>
      <value name="allowed_clients">127.0.0.1</value>

      <value name="environment">
        <value name="HOSTNAME">$HOSTNAME</value>
        <value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <value name="TMP">/tmp</value>
        <value name="TMPDIR">/tmp</value>
        <value name="TEMP">/tmp</value>
        <value name="OSTYPE">$OSTYPE</value>
        <value name="MACHTYPE">$MACHTYPE</value>
        <value name="MALLOC_CHECK_">2</value>
      </value>
    </section>
  </workers>
</configuration>

Server config :

Intel Corei5 (Lynnfield) 4x 2.66+ GHz 8 Mo L2 - FSB 1333 MHz
64 bits
8 Go DDR3
Intel SSD X25-M 2x 80 Go RAID   0/1
GigaEthernet

If anyone have an idea :slight_smile:

It’s very strange because it’s instant and random whereas it would be
timedout all the time :confused:

Thank’s in advance.

Posted at Nginx Forum:

This should give you some answers:

$ cat /var/log/nginx/error.log

Best regards,
Piotr S. < [email protected] >

150 children is quite a lot. Sure you need that? :slight_smile:

Usually using 5 is quiet enough.

Regards,
Joe

2010/5/30 Michael S. [email protected]

2010/5/29 Joe [email protected]:

Usually using 5 is quiet enough.

either you have a well optimized website with a 99% cache ratio or you
don’t have traffic. In the real world, there is so much web apps which
consume a lot of resources and for thoses you’ll need much more than 5
children to handle them.

So yes, I agree with you Loyet.
Currently using apc combined with the fpm spawn.
Frequently fpm restarted, and might be raising up the children in the
future.
We should take care about it very well, based on daily usage experience,
to
get the good nginx and fpm tune configuration.

Regards,
Joe

2010/5/30 Jérôme Loyet [email protected]

I’ve just allow error logs. I will wait a new 502 Bad Gateway.
And no, I don’t get 150 children but generally about 10.

Thank’s !

Posted at Nginx Forum:

Yup. General rule of thumb is roughly how many concurrent php requests
you expect at peak time. Hopefully your code runs fast enough to spend
less than 1 second processing. Cause if the children are busy it will
block until one opens up or until nginx’s configured timeout hits.

Yes Mike, that’s a very logic things.

Regards,
Joe

Thank’s for your responses.

At peak time I have about 15 children exept if I’m getting attacked :confused:
My server is the object of a lot of DoS attacks and it is possible to
get 100 children.
But you have to know that this server is use for a website but it is
also the infos server of my 30 servers of downloads.
I have Suhosin with APC enabled and my website has a SQL cache created
by me.

I should decrease the value of children ?

And sorry but after activating error logs I have not managed to
reproduce the problem and logs are still empty.

Posted at Nginx Forum:

is you’re getting attacked you shouldn’t be allowing it to execute php
you should be blocking it :slight_smile:

Nginx limit_conn does that, no ?
Look at my Nginx config :slight_smile:

Posted at Nginx Forum:

I’ve already limit_conn in my config file.

Active connections: 51
server accepts handled requests
 5634 5634 6587
Reading: 1 Writing: 1 Waiting: 49

Posted at Nginx Forum:

2010/5/29 TheCrach [email protected]:

Thank’s for your responses.

At peak time I have about 15 children exept if I’m getting attacked :confused:
My server is the object of a lot of DoS attacks and it is possible to get 100 children.
But you have to know that this server is use for a website but it is also the infos server of my 30 servers of downloads.
I have Suhosin with APC enabled and my website has a SQL cache created by me.

I should decrease the value of children ?

it depends on your system. If your system is able to handle 150
concurent request so you’re fine with it. When you’re under DOS
attack, your server won’t crash. But if all children are working at
same time, new clients won’t be able to connect. In this case, the DOS
will be a success (from the point of view of the attacker).

You can look at some interesting nginx modules:
http://wiki.nginx.org/NginxHttpLimitZoneModule
http://wiki.nginx.org/NginxHttpLimitReqModule

I’ve just get a 502 Bad gateway. It’s very random but instant ( the
server do not take time to show the error message ).
This is strange because any logs appear in Nginx and PHP-FPM. And I’ve
decrease the children value to 64.
I don’t understand why this error appear !? For me, it seems that the
child is restarted at nginx loading :confused:

Posted at Nginx Forum:

Hummm … Always the same problem. Any ideas ?

Posted at Nginx Forum:

2010/5/29 TheCrach [email protected]:

http {
keepalive_requests 10;
output_buffers 2 32k;

{
break;
location ~ .php4(/|$) {
Fastcgi params :
fastcgi_read_timeout 45;
fastcgi_param DOCUMENT_URI $document_uri;
[/code]
60
-1

       <value name="pm">
           <value name="style">static</value>
           <value name="max_children">150</value>

are your sure you don’t get 150 concurent connections to FPM ?

I’ve just edit my PHP-FPM conf with :

                 <value name="emergency_restart_threshold">30</value>
    <value name="emergency_restart_interval">0s</value>
    <value name="process_control_timeout">5s</value>

I hope it will correct my problem :slight_smile:

Posted at Nginx Forum: