Leaking memory nginx 1.8.0

Detlef_R · July 7, 2015, 8:26am

Hello, need help with leaking memory
i installed fresh system , centos7.1 +directadmin + nginx1.8.0 +
mariadb5.5.41 + php54 & php56 both with php_fpm
the server it for webhosting and i have there 32gb for now have only 4
website on it, and after 1-2hours nginx take all memory when it come to
96%
it stop there and then all websites fall but nginx not, only after i
reset
nginx all back to normal for 1-2hours again.

how can i debug where the problem? or try find the problem? seems in
logs i
not see anything

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 8:51am

2015-07-07 14:25 GMT+08:00 evgeni22 [email protected]:

check your file descriptors’ limit, try increasing it

ulimit -n

evgeni22 · July 7, 2015, 10:02am

do you sure to increase? does not it will eat faster the memory?

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 3:04pm

$ su nginx --shell /bin/bash --command “ulimit -n”
4096

it not fix the problem

nginx 10062 0.0 44.5 14763332 14639972 ? S 14:56 0:00 nginx:
worker process
nginx 10063 0.0 44.5 14763332 14639964 ? S 14:56 0:00 nginx:
worker process
nginx 10064 0.1 44.5 14763332 14639984 ? S 14:56 0:00 nginx:
worker process
nginx 10065 0.1 44.5 14763332 14640880 ? S 14:56 0:00 nginx:
worker process

have you more advice?

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 7:13pm

$ nginx -V
nginx version: nginx/1.8.0
built by gcc 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC)
built with OpenSSL 1.0.1e-fips 11 Feb 2013
TLS SNI support enabled
configure arguments:
–add-module=…/modsecurity_nginx-2.8.0/nginx/modsecurity --user=nginx
–group=nginx --prefix=/usr --sbin-path=/usr/sbin
–conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid
–http-log-path=/var/log/nginx/access_log
–error-log-path=/var/log/nginx/error_log --with-ipv6
–without-mail_imap_module --without-mail_smtp_module
–with-http_ssl_module
–with-http_realip_module --with-http_stub_status_module
–with-http_gzip_static_module --with-http_dav_module --with-cc-opt=‘’-D
FD_SETSIZE=32768’’

i not sure yet but 4 last hours seems leak stoped after i stop
security_mod, can be the security_mod doing this leaking in nginx ?

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 7:04pm

On Tuesday 07 July 2015 02:25:41 evgeni22 wrote:

Could you provide “nginx -V” output?

wbr, Valentin V. Bartenev

evgeni22 · July 7, 2015, 7:19pm

On Tuesday 07 July 2015 13:12:42 evgeni22 wrote:

–error-log-path=/var/log/nginx/error_log --with-ipv6
–without-mail_imap_module --without-mail_smtp_module --with-http_ssl_module
–with-http_realip_module --with-http_stub_status_module
–with-http_gzip_static_module --with-http_dav_module --with-cc-opt=’’-D
FD_SETSIZE=32768’’

i not sure yet but 4 last hours seems leak stoped after i stop
security_mod, can be the security_mod doing this leaking in nginx ?

[…]

I’m sure it can.

wbr, Valentin V. Bartenev

evgeni22 · July 7, 2015, 9:48pm

Show us the nginx.conf, maybe you’re using some weird large value
somewhere.

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 9:37pm

ok need help, it not security_mod, it begin grow again.
when server begin activate, i mean users begin upload files or replace
files
the memory increase immediately and not going down after they finish.
now the server ate 12gb of ram, if i do reset to nginx it going down to
4.5gb ram.

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 11:12pm

security_mod disabled now. but the config is
nginx-modsecurity-enable.conf:

ModSecurityEnabled on;
ModSecurityConfig /etc/nginx/nginx-modsecurity.conf;

Did you disable the module just via configuration or did a full vanilla
nginx recompile, because the module could be still hooking in requests.

Also 2.8.0 seems a bit “oldish” since it’s been released more than a
year
ago Releases · SpiderLabs/ModSecurity · GitHub

Besides there are existing / confirmed memory leaks:

github.com/SpiderLabs/ModSecurity

nginx_refacorting leaks memory after reloading nginx

opened 03:42AM - 02 Jun 15 UTC

closed 01:38AM - 06 May 17 UTC

LanShi

Platform - Nginx TBF by libmodsec

Hi, I test Nginx1.8.0 with nginx_refactoring branch module .and find memory lea…k still exists after reloading nginx. This is my configurations: nginx.conf: http { ``` server { listen 80; server_name localhost; location / { ModSecurityEnabled on; ModSecurityConfig modsecurity.conf; proxy_pass http://localhost:8080; } } ``` } modsecurity.conf: SecDebugLog /var/log/modsec_debug.log SecDebugLogLevel 9 SecRuleEngine On include /usr/local/nginx/conf/modsecurity-crs/modsecurity_crs_10_setup.conf include /usr/local/nginx/conf/modsecurity-crs/base_rules/modsecurity_crs_41_sql_injection_attacks.conf I will be glad to provide other information if necessary. Thanks

… and this discussion also might be related:

rr

evgeni22 · July 7, 2015, 10:03pm

This my hardware server:
Processor Intel Xeon E3 1225v2
Cores/Threads 4 cores/ 4 threads
Frequency 3.2 GHz+
RAM 32GB DDR3
Disks 3 x 120 GB SSD

=========================================
the config is nginx.conf:

#user nginx;

The number of worker processes is changed automatically by

CustomBuild,
according to the number of CPU core$
worker_processes 4;
pid /var/run/nginx.pid;
error_log /var/logs//nginx/error.log;

#error_log logs/error.log notice;
#error_log logs/error.log info;

events {
include /etc/nginx/nginx-events.conf;
}

http {
include /etc/nginx/mime.types;

access_log /var/log/nginx/access.log main;

# For user configurations not maintained by DirectAdmin. Empty by

default.
include /etc/nginx/nginx-includes.conf;

# Supplemental configuration
include /etc/nginx/nginx-modsecurity-enable.conf;
include /etc/nginx/nginx-defaults.conf;
include /etc/nginx/nginx-gzip.conf;
include /etc/nginx/directadmin-ips.conf;
include /etc/nginx/directadmin-settings.conf;
include /etc/nginx/nginx-vhosts.conf;
include /etc/nginx/directadmin-vhosts.conf;

}

config for nginx-defaults.conf:

default_type application/octet-stream;
tcp_nopush on;
tcp_nodelay on;
sendfile on;

log_format bytes ‘$bytes_sent $request_length’;

keepalive_timeout 2;
types_hash_max_size 2048;

disable_symlinks if_not_owner from=$document_root;

server_tokens off;

client_max_body_size 1024m;
client_body_timeout 3m;
client_body_buffer_size 128k;
client_header_timeout 20;
client_header_buffer_size 3m;
large_client_header_buffers 4 256k;
send_timeout 20;

proxy_connect_timeout 600s;
proxy_send_timeout 600s;
proxy_read_timeout 600s;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;

fastcgi_send_timeout 600s;
fastcgi_read_timeout 600s;

server_names_hash_bucket_size 128;
server_names_hash_max_size 10240;

ssl_dhparam /etc/nginx/ssl.crt/dhparams.pem;

Anti ddos

limit_conn_zone $binary_remote_addr zone=conn_limit_per_ip:10m;
limit_req_zone $binary_remote_addr zone=req_limit_per_ip:10m rate=5r/s;

================================
config for nginx-gzip.conf

gzip on;
gzip_static on;
gzip_disable “msie6”;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript
text/xml application/xml application$
gzip_buffers 16 8k;
gzip_min_length 10;
gzip_disable “MSIE [1-6].(?!.*SV1)”;

===================================
config for nginx/directadmin-ips.conf:

server {
listen xx.xx.xx.xx:80 default_server; // i hide the ip
server_name _;
root /home/admin/domains/sharedip;
index index.html index.htm index.php;

    include /usr/local/directadmin/data/users/admin/nginx_php.conf;

    # deny access to apache .htaccess files
    location ~ /\.ht
    {
            deny all;
    }

    include /etc/nginx/webapps.conf;

}

server {
listen xx.xx.xx.xx:443 default_server; // i hide the ip
server_name _;

    ssl                  on;
    ssl_certificate      /etc/nginx/ssl.crt/server.crt.combined;
    ssl_certificate_key  /etc/nginx/ssl.key/server.key;

    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout  5m;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    root /home/admin/domains/sharedip;
    index index.html index.htm index.php;

    include /usr/local/directadmin/data/users/admin/nginx_php.conf;

    # deny access to apache .htaccess files
    location ~ /\.ht
    {
            deny all;
    }

    include /etc/nginx/webapps.conf;

}

===========================
config for directadmin-settings.conf:

empty

===========================
config for nginx-vhosts.conf

server {
    listen       xx.xx.xx.x:80; // i hide the ip
    listen       127.0.0.1:80;
    #listen       [::1]:80;
    server_name  $hostname xx.xx.xx.xx; / i hide the ip

    root   /var/www/html;
    index  index.html index.htm index.php;

    #Support UserDir (~/user/)
    location ~^/~(?<userdir_user>.+?)(?<userdir_uri>/.*)?$ {
            alias /home/$userdir_user/public_html$userdir_uri;
            index index.html index.htm index.php;
            autoindex on;

            location ~ \.php$ {
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                include /etc/nginx/fastcgi_params;
                fastcgi_index index.php;

                #try_files does not work after alias directive
                if (!-f $request_filename) {
                    return 404;
                }

                fastcgi_param DOCUMENT_ROOT

/home/$userdir_user/public_html;
fastcgi_param SCRIPT_FILENAME $request_filename;
fastcgi_pass
unix:/usr/local/php54/sockets/$userdir_user.sock;
}
}

    # Pass all .php files onto a php-fpm/php-fcgi server.
    location ~ \.php$ {
            try_files $uri =404;

            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            include /etc/nginx/fastcgi_params;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME

$document_root$fastcgi_script_name;
fastcgi_pass unix:/usr/local/php54/sockets/webapps.sock;
}

    include /etc/nginx/nginx-info.conf;
    include /etc/nginx/webapps.conf;
}

HTTPS server

#
server {
    listen       xx.xx.xx.xx:443 ssl; / i hide the ip
    listen       127.0.0.1:443 ssl;
    #listen       [::1]:443 ssl;
    server_name  $hostname xx.xx.xx.xx; / i hide the ip

    ssl                     on;
    ssl_certificate         /etc/nginx/ssl.crt/server.crt.combined;
    ssl_certificate_key     /etc/nginx/ssl.key/server.key;

    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout  10m;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers   HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers   on;

    root   /var/www/html;
    index  index.html index.htm index.php;

    #Support UserDir (~/user/)
    location ~^/~(?<userdir_user>.+?)(?<userdir_uri>/.*)?$ {
            alias /home/$userdir_user/private_html$userdir_uri;
            index index.html index.htm index.php;
            autoindex on;

            location ~ \.php$ {
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                include /etc/nginx/fastcgi_params;
                fastcgi_index index.php;

                #try_files does not work after alias directive
                if (!-f $request_filename) {
                    return 404;
                }

                fastcgi_param DOCUMENT_ROOT

/home/$userdir_user/private_html;
fastcgi_param SCRIPT_FILENAME $request_filename;
fastcgi_pass
unix:/usr/local/php54/sockets/$userdir_user.sock;
}
}

    # Pass all .php files onto a php-fpm/php-fcgi server.
    location ~ \.php$ {
            try_files $uri =404;

            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            include /etc/nginx/fastcgi_params;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME

$document_root$fastcgi_script_name;
fastcgi_pass unix:/usr/local/php54/sockets/webapps.sock;
}

    include /etc/nginx/nginx-info.conf;
    include /etc/nginx/webapps.ssl.conf;
}

====================================
config for directadmin-vhosts.conf:

for individual clients i think not matter it empty there.

=================================
config for: nginx/nginx-info.conf

location /nginx_status {
# Enable nginx status page
stub_status on;

# Disable status page logging in access_log
access_log off;

# Allow access from 127.0.0.1
allow 127.0.0.1;

# Deny all the other connections
deny all;

}

security_mod disabled now. but the config is
nginx-modsecurity-enable.conf:

ModSecurityEnabled on;
ModSecurityConfig /etc/nginx/nginx-modsecurity.conf;

============================

Posted at Nginx Forum:

evgeni22 · July 7, 2015, 11:21pm

nginx recompile, because the module could be still hooking in requests.

Did read a bit further in the thread and could even confirm this:

“ModSecurity using resources even if it is disabled”

github.com/SpiderLabs/ModSecurity

SecRuleEngine off - still performance numbers drop

opened 05:11AM - 13 Jan 14 UTC

closed 02:23PM - 09 May 17 UTC

mitha

Platform - Nginx TBF by libmodsec

Hi modsecurity version - 2.7.5 nginx versoin - 1.4.2 owasp rules set - 2.2.8 …I am using modsecurity as WAF inline with nginx serving as reverse proxy. I am trying to measure the performance of nginx with mod security with SecRuleEngine - off,detection,on and without modsecurity. What i find is as soon as modsecurity is introduced in nginx.conf ,though SecRuleEngine is set to off,the performance - transaction per second - TPS drops.There is no significant difference between SecRuleEngine being set to on/off/detection in TPS measurement. I could see in the modsecurity debug log that the transactions are not processed # My question is : What i dont understand is,even though the transactions are not processed in modsecurity when SecRuleEngine is off why the performance should drop when it is introduced in nginx? configuration TPS nginx only 3594 nginx+modsec+SecRuleEngine-off 1399 nginx+modsec+SecRuleEngine-detection 342 nginx+modsec+SecRuleEngine-on 580 Load generator/Stress tool used - weighttp(WG) Configuration: The following configuration is kept on for all the below given tests total request - 500000 connection rate - 100/s keep alive - on file size fetched - 1kb cpu - 1 All the fetches are genuine ,No threats were injected VM size - small nginx running process - 1 each master,worker,cache manager process

which leads to “nginx: Returning 500 if SecRuleEngine is set to Off”

which is kind of fixed in

tagged as 2.9.0-rc2 so way after the version you use.

rr

evgeni22 · July 8, 2015, 8:22am

“nginx recompile, because the module could be still hooking in
requests.”

security_mod disabled it not tracker after any domains on the server it
cant
hooking requersts if it not tracking after domains
becuse memory leaking only when clients do traffic with domains.

i for sure now it not security_mod problem and about version i use
mod_security it last one
Current rules version 1.11 (Latest version)
CWAF plugin version 2.11 (Latest version)

i thinking of php-fpm problems with nginx, i google a little and saw
people
complain about leaking in nginx who have php-fpm

Posted at Nginx Forum:

evgeni22 · July 8, 2015, 8:57am

wondering if it good config in php-fpm

pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 60
pm.max_requests = 1000

?

Posted at Nginx Forum:

evgeni22 · July 8, 2015, 2:47pm

i only not understand why pm = ondemand doing the leaking, does algorithem
in nginx not stable or it php issue ?

It contradicts your previous mails about nginx processes being the leak
point.

php(-fpm) and nginx are separate pieces of software (just communicating
via
fastcgi protocol) so if the php leaks memory it’s a problem on php side
(is
also very valid considering that not all of the (third-party) php
extensions
are memory-friendly).

rr

evgeni22 · July 8, 2015, 9:45am

ok the leaking fixed , i will past here for database for other users:

that was in my php-fpm.conf

pm = ondemand
pm.max_children = 10
pm.process_idle_timeout = 60
pm.max_requests = 1000

=============================
after i change it to this conf the leaking stoped.

pm = dynamic
pm.max_children = 10
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 500

i only not understand why pm = ondemand doing the leaking, does
algorithem
in nginx not stable or it php issue ?

Posted at Nginx Forum:

evgeni22 · July 8, 2015, 4:34pm

Hi,
On 07/08/2015 09:44 AM, evgeni22 wrote:

after i change it to this conf the leaking stoped.

pm = dynamic
pm.max_children = 10
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 500
Very probably that last setting helped you, from the documentation:

pm.max_requests int
The number of requests each child process should execute before
respawning. This can be useful to work around memory leaks in 3rd party
libraries. For endless request processing specify ‘0’. Equivalent to
PHP_FCGI_MAX_REQUESTS. Default value: 0.

IMHO It’s a bandaid, but works.