Detect suspicious activity with nginx

Server NGINX
1

Hi guys

We are looking for a way to detect suspicious activity on high-traffic
websites. Parsing log files is not good option here, our current nginx
config generates around 90G of logs for around 412K http requests each
days.

We are looking to use nginx to detect suspicious activity and generate
precise log when it happens for post-processing.

Some tools we are looking for would be something like

  • Detect IPs which accessed /uri1/ X times without accessing other URI
    in a period of time Y.

  • Detect IPs that are indexing our site by accessing sequential uris
    like /uri123, /uri124, /uri125, …

We are using load balancing services (haproxy), we enabled realip module
in nginx, we need something that can work with it.

If you have any pointers / ideas / module names that could help us,
please let me know.

Have a good day

Max

2

Hi Max,

In my opinion you dont want to rely on nginx to do the analytics simply
to
log suspicious activity but rather need to look at a better
log analyzing solution. Have you checked out splunk? It is a very
powerful
log analyzer that will allow you to more intelligently parse the logs
and
has a free licence.

Rami

3

I would also look into some kind of blacklist, like
spam/drone/proxy/etc, to compare IP addresses that are suspicious and
kindly redirect them to /dev/null

-C

4

You can try ides from GitHub - znarf/bouncer: A PHP library to log and analyse HTTP traffic, throttle and block suspicious agents. and move then
into
Lua or Perl.