I’m trying to make POST analyzer module. To have ability with SPAM
detection for passing throw traffic, for this I need to have
request_body in subrequest, for that I took agentzh / nginx-eval-module
and after reading this posts made small changes (please find it below)
on small tests it working fine, but on something with big POSTs it
loosing information, could somebody help me with understanding how
better fix this.
I’m trying to make POST analyzer module. To have ability with SPAM detection
for passing throw traffic, for this I need to have request_body in
subrequest, for that I took agentzh / nginx-eval-module
and after reading this posts made small changes (please find it below)
on small tests it working fine, but on something with big POSTs it loosing
information, could somebody help me with understanding how better fix this.
I am afraid you’re using ngx_http_read_client_request_body in a wrong
way. When the request body is big enough to exceed the preread part of
the request headers or the network is bad enough, you’ll surely loose
data in the request body.
Nginx works in a streaming and non-blocking manner, so you have to wait for the ngx_http_read_client_request_body fully complete (not
just when this function returns but when your ngx_http_stub is called)
before continuing with your rewrite-phase handler. So it’s totally
wrong to do nothing in your ngx_http_stub callback. It should be the
key part for your logic.
Please take a peek at the ngx_form_input module to have a live demo for
this:
http://github.com/calio/form-input-nginx-module
It was originally written by one of my intern students and now
maintained by me
This module was from which I begun. Originally I saved all input data into
variable and tried to pass it into fastcgi subrequest, unfortunately when
I’m passing this variable in subrequest request_body is already empty,
How did you pass this variable into your subrequest? It looks weird to
me
because subrequest have empty request_body, that’s was the reason why I’m
trying to pass request_body to subrequest. May be people here tried other
variants or have some ideas to have good quality of post analysis
internally?
I don’t think it’s a good idea to use subrequests here just for post
body analysis. A rewrite phase or post access phase handler seems
rather sufficient to me.
Passing request bodies to a subrequest is as simple as passing
r->request_body->bufs to sr->request_body->bufs.
the request headers or the network is bad enough, you’ll surely loose
http://github.com/calio/form-input-nginx-module
This module was from which I begun. Originally I saved all input data
into variable and tried to pass it into fastcgi subrequest,
unfortunately when I’m passing this variable in subrequest request_body
is already empty, because subrequest have empty request_body, that’s was
the reason why I’m trying to pass request_body to subrequest. May be
people here tried other variants or have some ideas to have good quality
of post analysis internally?
It was originally written by one of my intern students and now
maintained by me
location = /check-spam {
internal;
include spam_fastcgi.conf;
fastcgi_pass unix:/var/run/nginx/cgiwrap-dispatch.sock;
}
location / {
client_max_body_size 100k;
client_body_buffer_size 100k;
lua_need_request_body on;
rewrite_by_lua '
if ngx.var.request_method ~= "POST" then
return
end
local res = ngx.location.capture(
"/check-spam",
{ method = "POST", body = ngx.var.request_body }
)
if res.body == "SPAM" then
return ngx.redirect("/terms-of-use.html")
end
';
# ... your normal fastcgi_pass/proxy_pass/etc settings
# go here ...
}
Please note that you need the git HEAD of the “master” branch of the
ngx_lua git repos:
https://github.com/chaoslawful/lua-nginx-module
I have some foreign spam analyzer which I’m running like fastcgi server
written in perl.
I tried to use $request_body as fastsgi_param arg, it was empty;
Wrote module which creates $post_body var in rewrite stage and with echo
$post_body it works, but on subrequest it gives empty body (I guess because
it not passing POST into subrequest)
maybe somebody have any ideas how to realize it other way, or may be how to
get POST in subrequest?
If you have further issues, feel free to report here
This module was from which I begun. Originally I saved all input data into
variable and tried to pass it into fastcgi subrequest, unfortunately when
I’m passing this variable in subrequest request_body is already empty,
How did you pass this variable into your subrequest? It looks weird to me
Let me show one variant of nginx.conf’s:
location / {
include proxy.conf;
eval_subrequest_in_memory off;
eval_override_content_type text/plain;
eval $res {
include spam_fastcgi.conf;
if ( $request_method = POST) {
fastcgi_pass unix:/var/run/nginx/cgiwrap-dispatch.sock;
}
}
if ( $res = SPAM) {
rewrite ^ /terms-of-use.html redirect;
}
}
I have some foreign spam analyzer which I’m running like fastcgi server
written in perl.
I tried to use $request_body as fastsgi_param arg, it was empty;
Wrote module which creates $post_body var in rewrite stage and with echo
$post_body it works, but on subrequest it gives empty body (I guess
because it not passing POST into subrequest)
maybe somebody have any ideas how to realize it other way, or may be how
to get POST in subrequest?
Thank you for quick response, and I afraid of my stupid questions about
embedded lua but :
I’m using current development nginx-0.9.3 (my internals need huge
table ~350k domain records for 2 different nginx parameters) and doable
in current version.
when I’m just including rewrite_by_lua with simple return, I’m
loosing all headers from parent cite, please help me with best solution
for that.
Thank you for quick response, and I afraid of my stupid questions about
embedded lua but :
I’m using current development nginx-0.9.3 (my internals need huge table
~350k domain records for 2 different nginx parameters) and doable in current
version.
I’ve just confirmed that ngx_lua’s test suite is passing completely
with nginx 0.9.3
when I’m just including rewrite_by_lua with simple return, I’m loosing
all headers from parent cite, please help me with best solution for that.
I’m not sure what you’re talking about here. Can you provide a minimal
config sample for this issue?
why you prefer NDK to clear nginx environment?
NDK just provides the “set_var” mechanism for ngx_lua. What do you
mean by “clearing nginx environment”?
Please also note that due to a famous regression regarding rewrite
phase handling in nginx 0.8.42 ~ 0.8.53, you need at least nginx
0.8.54+ to get rewrite_by_lua work
HTTP/1.1 200 OK
Server: nginx/0.9.3
Date: Fri, 07 Jan 2011 20:10:38 GMT
Content-Type: text/html
Connection: close
Accept-Ranges: bytes
Via: 1.1 netcache-c (NetCache NetApp/5.6.2R1D19)
==================
location / {
include proxy.conf;
}
why you prefer NDK to clear nginx environment?
NDK just provides the “set_var” mechanism for ngx_lua. What do you
mean by “clearing nginx environment”?
I mean if you using just this function may be better to include it into
your module
NDK patchnig nginx and it not clear, because you actually never need it.
I mean if you using just this function may be better to include it into your
module
NDK patchnig nginx and it not clear, because you actually never need it.
The latest NDK (pre-)release no longer patches the nginx core:
I’ve installed https://github.com/chaoslawful/lua-nginx-module/tarball/v0.1.2rc3
with this config
location / {
rewrite_by_lua ’
return ngx.redirect(“/terms-of-use.html”)
';
include proxy.conf;
}
and it doesnt rewrites anything, just working like without this line.
is it correct or something which we need to fix next?
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
