Hello All!
I am attempting to do stress testing of a custom proxy server.
Basically,
the scenario looks somewhat like the following:
performance testing software (client) —> proxy server —> nginx
(server).
The nginx server is an Ubuntu 14.04 install. I have done a small amount
of
reading on tuning nginx for performance, so I have increased the number
of
worker_processes to fit my cpu, set worker_rlimit_nofile very high, told
nginx to use epoll, sendfile, tcp_nopush, and tcp_nodelay. I have also
increased the number of worker_connections to 4000, though in my case I
did
not see any change from its previous default value of 768. The nginx
server
is serving up static content only.
Under line-speed circumstances, everything works well and the nginx
server
appears to be able to match (if not exceed) the abilities of the proxy
server. Performance issues and bugs worked out of the proxy server, I
decided I wanted to attempt to simulate an artificial latency. To do
this,
I decided to use the linux tc plugin netem. I used the following command
to
create an artificial latency between the nginx server and the proxy
server:
tc qdisc add dev eth0 root netem delay 30ms 20ms distribution normal
Prior to running this command, I could fully saturate the 1 gbps
connection
rate of the nginx server. Now that there is latency, it would make sense
that the connection rate would diminish. It dropped to a maximum of 450
mbps. To increase the speed, therefore, I decided to increase the number
of
connections. To my surprise, this had no effect. The speed stayed at 450
mbps. I don’t get any error messages on either end, the connections just
take longer to resolve. However, the load level on the nginx server does
not change.
Thinking something about the way netem worked had caused the line speed
to
drop, I used the tool iperf to test the connection speed between the
client
and the server, I got the full 1gbps.
At this point, I bypassed the proxy software thinking it was the
problem.
This did leave the hardware with the proxy server on it routing the
traffic. It is linux based. Bypassing the proxy software did not change
the
performance.
Does anyone have any suggestions on how I can solve this? I would have
expected an increase in the number of connections to yield an increase
in
speed, but this was not the result. It seems with the artificial latency
nginx is only able to handle approximately 3,000 requests per second,
where
without the latency I saw over 7,000.
Thank you all for your help, and I will try to provide more information
if
that will help anyone help me solve this problem.