Ngx_lua v0.1.5: ability to capture multiple parallel subrequests

Hi, folks!

I’m pleased to announce that ngx_lua v0.1.5 is now released. You can
download the release tarball from the download page:

https://github.com/chaoslawful/lua-nginx-module/downloads

This version of the ngx_lua module implements the
ngx.location.capture_multi method that can issue multiple nginx
subrequests at the same time, capture their responses along the way,
and then return the results in the same order that they’re specified
in the arguments.

Here is a small example:

location ~ ‘^/echo/(\w+)$’ {
echo -n $1;
}

location = /main {
content_by_lua ’
local res1, res2, res3 =
ngx.location.capture_multi{
{ “/echo/sun” },
{ “/echo/moon” },
{ “/echo/earth” },
}
ngx.say(res1.body)
ngx.say(res2.body)
ngx.say(res3.body)
';
}

then accessing /main gives

sun
moon
earth

and those three subrequests, GET /echo/sun, GET /echo/moon, and GET
/echo/earth, were issued at the same time.

Below is a more complicated example that fires 2 concurrent MySQL
queries to calculate the count of dogs and the count cats,
respectively, at the same time, and finally output the sum of these 2
counts, i.e., the total count of all the animals.

First of all, we define the remote mysql server upstream in the main
http block:

upstream mysql_node {
drizzle_server 10.32.136.5:3306
user=monty password=some_pass dbname=main protocol=mysql;
drizzle_keepalive max=100 mode=single overflow=ignore;
}

then we define a general-purpose internal location to proxy SQL
queries to remote MySQL node in our default virtual host server block:

location = /mysql {
internal;
drizzle_query $echo_request_body;
drizzle_pass mysql_node;
rds_json on;
}

after that, we define our main location that does the job:

location /main {
content_by_lua ’
local opts1 = {
method = ngx.HTTP_POST,
body = “select sum(1) cnt from cats”,
}
local opts2 = {
method = ngx.HTTP_POST,
body = “select sum(1) cnt from dogs”,
}
local res1, res2 =
ngx.location.capture_multi{
{ “/mysql”, opts1 },
{ “/mysql”, opts2 }
}

local yajl = require(“yajl”)
local cats = yajl.to_value(res1.body)
local dogs = yajl.to_value(res2.body)

local total = cats[1].cnt + dogs[1].cnt
ngx.say("for total “, total, " animals.”)
';
}

For brevity, I’ve omitted all the error handling code like checking if
res1.status is 200 and validating cats.errcode has nil values :wink:

The nginx modules ngx_drizzle, ngx_rds_json, and the lua-yajl Lua
library are also used in the sample above.

This example can be trivially extended to run tens or even more sql
queries to different MySQL machines and combined with more PostgreSQL
database servers. All in parallel and all are 100% non-blocking on
network traffic.

This v0.1.5 version of ngx_lua also contains

* optimizations on subrequest buffer management and thus saving

much memory when using ngx.location.capture (and
ngx.location.capture_multi).

* fixes of the build system that we no longer require OpenSSL and

ngx_lua no longer crash when used with statically linked OpenSSL.
Thanks Marcus C. and Vladislav Manchev.

* fixes of the Lua code cache handling that we no longer clear

pre-loaded standard Lua packages like “table” and “string” when the
Lua code cache is off.

The ngx_lua module embeds the Lua/LuaJIT interpreter into the nginx
core and integrates the powerful Lua threads (aka Lua coroutines) into
the nginx event model by means of nginx subrequests.

You can get the latest source code and full documentation from the
ngx_lua’s project page:

GitHub - openresty/lua-nginx-module: Embed the Power of Lua into NGINX HTTP servers

Happy Lua and nginx.conf hacking!

Cheers,
-agentzh

On Wed, Feb 9, 2011 at 6:34 PM, Marcus C. [email protected] wrote:

Hi,

Nice work guys!

Thanks! :smiley:

Is it also possible to pass an array of requests and get an array of
responses, or otherwise send requests in parallel when you don’t know in
advance how many subrequests you want to issue?

Sure! For instance,

-- construct the requests table
local reqs = {}
table.insert(reqs, { "/mysql" })
table.insert(reqs, { "/postgres" })
table.insert(reqs, { "/redis" })
table.insert(reqs, { "/memcached" })

-- issue all the requests at once and wait when they all return
local resps = { ngx.location.capture_multi(reqs) }

-- loop over the responses table
for i, resp in ipairs(resps) do
     -- process the response table "resp"
end

Well, it’s mostly a feature on the Lua language level :wink:

Looking at the documentation, it doesn’t appear that this is currently
possible.

I’ll add a note for this to ngx_lua’s documentation :slight_smile:

I’d like to get into using Lua on Nginx for something I’m working on right
now that issues lots of subrequests, but I don’t know how many I need to
issue in advance.

Yeah, that’s understandable :slight_smile:

BTW, I’ve cc’d the nginx mailing list for reference :slight_smile:

Cheers,
-agentzh

Hi,

On 10/02/2011 07:30, agentzh wrote:

 table.insert(reqs, { "/memcached" })

Great - thanks for the example. I’m not overly familiar with Lua yet,
but hopefully will be more so soon.

Cheers,

Marcus.

Thanks for sharing so many of your Nginx addons, Nginx scripting is so
exciting. I read
your 2 slides about Nginx scripting. Very intersting, are there some
more examples or informations
about Nginx scripting?

I try out your example, you subrequest on a redis location. I use the
redis2 addon, but i
allways get in the first line the bytes for my redis request. Is there a
solution hide this information?
So it would be possible to use redis2 for caching whole HTML pages.

Have a nice day agentzh!

Best regards

Alexander

I’ve updated on one of my production servers mod-lua to 0.1.5, and nginx
memory still growing.
My config is still the same
location / {
include forward_proxy.conf;

                 location = /check-spam {
                         internal;

                         include spam_fastcgi.conf;
                 }

                 lua_need_request_body on;

                 rewrite_by_lua '
                   if ngx.var.request_method ~= "POST" then
                     return
                   end
                   local res = ngx.location.capture( "/check-spam",

{ method = ngx.HTTP_POST, body = ngx.var.request_body })
local a=string.sub(res.body,0,4)==“SPAM”
if a then
return ngx.redirect(“/terms-of-use.html”)
end’;
}

2011/02/10 12:36:00 [alert] 3090#0: *21747555 zero size buf in output
t:1 r:0 f:0 0000000006D42970 0000000006D42970-0000000006D42970
0000000000000000 0-0 while sending request to upstream, client:
10.1.24.20, server: hss, request: “POST /xxx/xxx.aspx HTTP/1.1”,
upstream: “http://x.x.x.x/xxx/xxx.aspx”, host: “xxx.com
2011/02/10 12:36:07 [alert] 3085#0: *21749071 sendfile() failed (9: Bad
file descriptor) while sending request to upstream, client: 10.1.16.17,
server: hss, request: “POST /xxx/xxx.aspx HTTP/1.1”, upstream:
http://x.x.x.x/xxx/xxx.aspx”, host: “xxx.com

On 02/10/2011 02:38 PM, Roman V. wrote:

                }
                  if a then
                    return ngx.redirect("/terms-of-use.html")
                  end';
        }

Forgot to say that below is my error log with 2 very frequent messages:

On Thu, Feb 10, 2011 at 6:26 PM, Alexander K. [email protected]
wrote:

Thanks for sharing so many of your Nginx addons, Nginx scripting is so
exciting. I read
your 2 slides about Nginx scripting. Very intersting, are there some more
examples or informations
about Nginx scripting?

Maybe my blog count? :wink:

http://agentzh.blogspot.com/

I do have the plan of writing a bunch of (detailed) tutorials on this
this year.

I try out your example, you subrequest on a redis location. I use the redis2
addon, but i
allways get in the first line the bytes for my redis request. Is there a
solution hide this information?

The ngx_redis2 module returns the raw TCP response from the redis
server. It’s recommended to use my lua-redis-parser module (written in
pure C) to parse these responses into lua data structure:

https://github.com/agentzh/lua-redis-parser

If you only want to use the “get” redis command, you can try out the
ngx_redis module here:

http://wiki.nginx.org/HttpRedis

It returns the parsed content part of the redis response because only
“get” is needed to implement.

So it would be possible to use redis2 for caching whole HTML pages.

Right, using ngx_srcache + ngx_lua + lua-redis-parser. See ngx_srcache
for details:

http://github.com/agentzh/srcache-nginx-module

It’s much like Apache2’s mod_cache, but only support silly response
body caching (for now) and is 100% non-blocking. I’re already using
ngx_srcache + ngx_memc to cache responses via memcached in production.

Have a nice day agentzh!

Thanks! :slight_smile: The same to you :wink:

Cheers,
-agentzh

On Fri, Feb 11, 2011 at 6:41 AM, Roman V. [email protected]
wrote:

Forgot to say that below is my error log with 2 very frequent messages:

2011/02/10 12:36:00 [alert] 3090#0: *21747555 zero size buf in output t:1
r:0 f:0 0000000006D42970 0000000006D42970-0000000006D42970 0000000000000000
0-0 while sending request to upstream, client: 10.1.24.20, server: hss,
request: “POST /xxx/xxx.asmx HTTP/1.1”, upstream:
http://x.x.x.x/xxx/xxx.asmx”, host: “xxx.com

What are your client_max_body_size and client_body_buffer_size settings?

It seems that your request body has been tainted which could happen
when client_max_body_size > client_body_buffer_size and the body is
big enough buffered to a disk file.

But I cannot reproduce any leaks or thse zero size buf errors on my
side or on our production servers.

It’ll be great if you provide the following additional information:

  1. your lua library’s version (or any 3rd-party patches applied by
    your OS vendor),
  2. your nginx -V output (or any 3rd-party patches applied by your OS
    vendor),
  3. the precise memory change curve of your nginx worker processes over
    time,
  4. the Lua function call collectgarbage(“count”) output when you think
    your nginx is leaking,
  5. sample requests that can easily reproduce the leak on other
    environment, like on my laptop.

Cheers,
-agentzh

agentzh,

I was wondering if you had considered having a capture mode that used a
callback rather than coroutines? In really high traffic servers, the
coroutines seem to eat a good bit of memory.

An example use could be:

Define a callback

function mycallback(res)
do_stuff_with_res
end

Then later…

ngx.location.capture("/some_other_location", mycallback)

You could also pass closures as well:

x = something_I_care_about
callback = function (res) return someothercallback(res, x) end
ngx.location.capture("/some_other_location", callback)

My Lua is a little rusty, but you get the idea.


Brian A.

location = /t-get-lua-mem-use {
content_by_lua ’
ngx.say("lua use " … collectgarbage(“count”) … “KB”)
';
}

maybe you can use this to got how many memory used by lua.
and then find out what the problem is.

On Fri, 2011-02-11 at 11:55 +0800, agentzh wrote:

On Fri, Feb 11, 2011 at 2:12 AM, Akins, Brian [email protected] wrote:

agentzh,

I was wondering if you had considered having a capture mode that used a
callback rather than coroutines? In really high traffic servers, the
coroutines seem to eat a good bit of memory.

Have you tried getting Lua to garbage collect before yielding? That
might help.

standard modules (via patches) to release buffers at the end of the
subrequest, rather than main request.

That sounds a good fix…

On Fri, Feb 11, 2011 at 2:12 AM, Akins, Brian [email protected]
wrote:

agentzh,

I was wondering if you had considered having a capture mode that used a
callback rather than coroutines? In really high traffic servers, the
coroutines seem to eat a good bit of memory.

Have you tried LuaJIT 2.0? It saves 25+% of the total RAM used by our
nginx worker processes in our business, compared to the standard Lua
5.1 interpreter. Also the latter’s coroutine implementation is
suboptimal.

Another issue is that most of the upstream modules do not release its
output bufs as early as possible in the context of subrequests, they
usually rely on the nginx memory pool to release all those bufs when
the pool is destroyed at the end of the main request, which is quite
unacceptable. We’ll fix our upstream modules and possibly other
standard modules (via patches) to release buffers at the end of the
subrequest, rather than main request.

Technically speaking, callbacks won’t save memory, we still need to
save all your Lua context such that you can get access to data in the
outer context in your Lua callback, or it’ll be useless :slight_smile:

Cheers,
-agentzh

On Fri, Feb 11, 2011 at 5:11 PM, Justin Cormack
[email protected] wrote:

Have you tried getting Lua to garbage collect before yielding? That
might help.

Forcing a full garbage collection cycle will block the nginx worker
for a while because nginx is single threaded and the Lua VM is shared
within the worker process. That’s also one of the reasons Java web
apps also freeze when JVM does a full GC cycle. So it only makes sense
when the system is really short of RAM resources :slight_smile:

Cheers,
-agentzh

About memory leak wont to apologize, my fault,.
Actually ngix memory usage increased but stable, and Valgrind not
showing any memory leaks.
location = /t-get-lua-mem-use {
content_by_lua ’
ngx.say("lua use " … collectgarbage(“count”) … “KB”)
';
}

Garbagecollector shows:
this shows range 40~100Kb.
Problem in perl script which realizing FCGI for spamassasin, we’ll fix
it.
Problem in log still exists:

On 02/10/2011 09:01 PM, agentzh wrote:

On Fri, Feb 11, 2011 at 6:41 AM, Roman V.[email protected] wrote:

Forgot to say that below is my error log with 2 very frequent messages:

2011/02/10 12:36:00 [alert] 3090#0: *21747555 zero size buf in output t:1
r:0 f:0 0000000006D42970 0000000006D42970-0000000006D42970 0000000000000000
0-0 while sending request to upstream, client: 10.1.24.20, server: hss,
request: “POST /xxx/xxx.asmx HTTP/1.1”, upstream:
http://x.x.x.x/xxx/xxx.asmx”, host: “xxx.com

even with commented out fastcgi request, so subrequest always return
404.
location / {
include forward_proxy.conf;

                 location = /check-spam {
                         internal;

                         #include spam_fastcgi.conf;
                 }

                 location = /t-get-lua-mem-use {
                         content_by_lua '
                                 ngx.say("lua use " ..

collectgarbage(“count”) … “KB”)
';
}

                 lua_need_request_body on;

                 rewrite_by_lua '
                   if ngx.var.request_method ~= "POST" then
                     return
                   end
                   local res = ngx.location.capture( "/check-spam",

{ method = ngx.HTTP_POST, body = ngx.var.request_body })
local a=string.sub(res.body,0,4)==“SPAM”
if a then
return
ngx.redirect(“pango group”)
end’;
}
but any way everything works fine, only logs full of “zero size buf in
output… request: POST…”

  1. your lua library’s version (or any 3rd-party patches applied by
    your OS vendor),

rpm -qa|grep lua

lua-5.1.4-0
lua-devel-5.1.4-0

  1. your nginx -V output (or any 3rd-party patches applied by your OS vendor),

nginx -V

nginx: nginx version: nginx/0.9.4
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
nginx: configure arguments: --prefix=/usr --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid
–lock-path=/var/lock/nginx.lock --user=www --group=www
–with-http_flv_module --with-http_gzip_static_module
–with-http_sub_module --http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/nginx/client/
–http-proxy-temp-path=/var/nginx/proxy/
–http-fastcgi-temp-path=/var/nginx/fcgi/ --with-http_stub_status_module
–add-module=/usr/src/redhat/SOURCES/af-headers
–add-module=/usr/src/redhat/SOURCES/echo-module
–add-module=/usr/src/redhat/SOURCES/ndk
–add-module=/usr/src/redhat/SOURCES/lua-module
–add-module=/usr/src/redhat/SOURCES/gunzip

  1. the precise memory change curve of your nginx worker processes over time,
  2. the Lua function call collectgarbage(“count”) output when you think
    your nginx is leaking,

showed upper

  1. sample requests that can easily reproduce the leak on other
    environment, like on my laptop.

I’m trying to find best way how to reproduce it, now it hard to
reproduce without load.

Also let me show my fastcgi parameters:
fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffers 4 256k;
fastcgi_buffer_size 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

Actually I don’t know which extra buffers here user, is it some
lua_buffers?

On Sat, Feb 12, 2011 at 4:39 AM, Roman V. [email protected]
wrote:

Actually I don’t know which extra buffers here user, is it some lua_buffers?

No :slight_smile:

What are your client_max_body_size and client_body_buffer_size
settings? Could you set them equal in your nginx.conf?

Also, please try out ngx_lua v0.1.6rc1 or git HEAD, which contains
some fixes regarding request body bufs and zero size buf alerts:

https://github.com/chaoslawful/lua-nginx-module/downloads

Thanks for your patience :slight_smile:

Cheers,
-agentzh

Please find memory usage graph attached.
Last graph with fastcgi turned off and killed perl.

On Tue, Aug 9, 2011 at 12:03 PM, agentzh [email protected] wrote:

I finally got this “zero size buf in output” alert while turning
lua_need_request_body on in the latest ngx_lua v0.2.1rc4 prerelease:

Sorry, I mean I’ve finally got this issue fixed :slight_smile: This bug will
appear when turning lua_need_request_body on and using
rewrite_by_lua* or access_by_lua* directives.

Regards,
-agentzh

On Fri, Feb 11, 2011 at 6:38 AM, Roman V. [email protected]
wrote:

     lua_need_request_body on;

2011/02/10 12:36:00 [alert] 3090#0: *21747555 zero size buf in output t:1
r:0 f:0 0000000006D42970 0000000006D42970-0000000006D42970 0000000000000000
0-0 while sending request to upstream, client: 10.1.24.20, server: hss,
request: “POST /xxx/xxx.aspx HTTP/1.1”, upstream:
http://x.x.x.x/xxx/xxx.aspx”, host: “xxx.com

I finally got this “zero size buf in output” alert while turning
lua_need_request_body on in the latest ngx_lua v0.2.1rc4 prerelease:

https://github.com/chaoslawful/lua-nginx-module/downloads

Thank you for reporting this!

Thanks!
-agentzh