High bandwidth configuration help

Server NGINX
1

Hi

how to tweak nginx for large file serving and maximizing the use of
available bandwidth?

I used to use lighttpd and was able to use 150mbit average at all times
with
lighttpd (writev backend selected) serving very large files t oalot of
people

But now i cant achievesame results

Can anyone point me to any config options to make large file serving
more
efficient, there arent many examples :frowning:

Thanks

2

Forgot to mention bellow is my nginx config

i can push about 80mbit average now but loads are above 6 (used to be
1-2
with lighttpd) this is after settings “sendfile off” switch, was only
pushing ~30mbit then :frowning:

heres my config (i replaced ips and hostnames for privacy reasons)

http://pastebin.com/m55a90c33

Thanks and hopefully someone can help me to migrate completely from
lighttpd
to nginx :slight_smile:

So any tips on how to optimise for alot of large concurent file
downloads
are welcome

!

On Sun, Feb 24, 2008 at 1:41 AM, nginx. mailinglist <

3

On 2/23/08, nginx. mailinglist [email protected] wrote:

Thanks and hopefully someone can help me to migrate completely from lighttpd
to nginx :slight_smile:

So any tips on how to optimise for alot of large concurent file downloads
are welcome

What is the OS you are running it on?

4

What is the configuration of /hdd1, is it capable of pushing that kind
of random IO load ?

How many workers are you running?

Cheers

Dave

5

On Sun, Feb 24, 2008 at 01:51:36AM +0000, nginx.mailinglist wrote:

Thanks and hopefully someone can help me to migrate completely from lighttpd
to nginx :slight_smile:

So any tips on how to optimise for alot of large concurent file downloads
are welcome

!

Set single worker process instead of 6. This is like single lighttpd
process.
It seems that your disk can not handle several concurrent accesses.
You may try to set worker_processes to 2 or 3 and look bandwidth.

6

Hi,

On Son 24.02.2008 01:51, nginx.mailinglist wrote:

Thanks and hopefully someone can help me to migrate completely from
lighttpd to nginx :slight_smile:

So any tips on how to optimise for alot of large concurent file
downloads are welcome

How looks your lighttpd conffile?

Have you changed anything else then the webserver?
Which nginx version do you use?
How was nginx compiled?

As eliott asked: Which OS do you use?

If you under linux maybe you can make a:

vmstat -d 1
and/or
vmstat 1

BR

Aleks

7

Hi

thanks for all the replies! i should have provided more details but I
was in
dispair and it was late at night, have clear head now after good sleep.

Ok heres info requested, the servers (i have another 6 pushing 120~mbit
average with lighttpd 1.4 compiled now, all servers same config)

OS: Suse 10.3 (minimal, i have barely anything installed and allmost
everythign compiled by me) http://pastebin.com/m41b5a046

Applications Running:
*nginx_0.5.35 (compiled http://pastebin.com/m6462475a ) used for file
downloads
*php_5.2.5 (compiled http://pastebin.com/m3c02ea6e , ~20 threads
started
with lighttps spawn fcgi), very short php scripts are run that use
X-Accell-Redirect to pass control to nginx of file serving
*lighttpd_1.5r2048 (compiled) used for file uploads, due to the progress
meter, fairly stable in production tho its from the svn
*custom php_cli socket deamons for inter server RPC, these are very
light

top http://pastebin.com/m2587b666
dstat http://pastebin.com/m2055efd4
df http://pastebin.com/m724e52a7
vmstat http://pastebin.com/m5cdc2f0b
**
heres lighttpd 1.4.18 config from other servers on network (they are
all
pretty much same just differing hosts) http://pastebin.com/m7b1af1e6

now the only thing that changed on this server is lighttpd 1.4.18 was
replaced by nginx, and now the request go thru nginx X-AcellRedirect not
lighttpd’s mod_secdownload

the php5 fcgi scripts are very small it check file exists etc, then just
adds a downlaod id into headers for nginx to pick up and passes control
onto
nginx the relevant bit is here > http://pastebin.com/m2bcbe7fd

basically the server is capable of pushing above 100mbit easily before,
now
the load is high, but this could be due to me setting high number of
worker
threads like Igor said and more php-cgi processes floating around

so to summarise I need to figure out how to tweak this so i can move all
the
servers from lighttpd to nginx for file downloads

Regards

8

On 2/24/08, nginx.mailinglist [email protected] wrote:

thanks for all the replies! i should have provided more details but I was in
dispair and it was late at night, have clear head now after good sleep.
[…]
top http://pastebin.com/m2587b666
dstat http://pastebin.com/m2055efd4
df http://pastebin.com/m724e52a7
vmstat http://pastebin.com/m5cdc2f0b

You need to run vmstat for a longer period of time (try “vmstat 1” and
let it run for a minute), but looking at that one line:

server11:/ # vmstat
procs -----------memory---------- —swap-- -----io---- -system–
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
0 5 84 49296 16044 1824108 0 0 17 11 14 11 3 18
43 36

It seems you are completely I/O-bound. The second column shows us that
five processes are blocking for I/O – ie., doing nothing except wait
for I/O.

I suggest you follow Igor’s advice and set:

worker_processes 1;

and see what happens then.

Alexander.

9

On Sun, Feb 24, 2008 at 09:47:02AM +0000, nginx.mailinglist wrote:

OS: Suse 10.3 (minimal, i have barely anything installed and allmost
*custom php_cli socket deamons for inter server RPC, these are very light

threads like Igor said and more php-cgi processes floating around

so to summarise I need to figure out how to tweak this so i can move all the
servers from lighttpd to nginx for file downloads

You should try to set
worker_processes 1;

LA in Linux is sum of processes that run or ready to run and processes
that
wait for disk I/O. LA 6 and “70.5%wa” means that all 6 nginx workers
wait
for disk I/O. Here is your bottleneck.

10

Hi see mrtg @ http://img402.imageshack.us/img402/8058/nginxwx0.png

before the drop was running lighttpd (consistently above 100mbit)

after nginx (80-90mbit)

:frowning:

the bandwdith is paid for but i cant utilise it which sucks

11

My recommendation for IO bound workloads is to use the Deadline
Scheduler rather than the anticipatory or CFQ scheduler

12

Use bonnie for test you storage space. And exucate “dmesg” (for Linux
OS)
and view any network problem.

On Sun, Feb 24, 2008 at 6:27 PM, Igor S. [email protected] wrote:

with lighttps spawn fcgi), very short php scripts are run that use
vmstat http://pastebin.com/m5cdc2f0b
the php5 fcgi scripts are very small it check file exists etc, then just
so to summarise I need to figure out how to tweak this so i can move all

–
Igor S.
Igor Sysoev

–
Please excuse my bad english…

С уважение,
Й. Георгиев.

WEB: http://gigavolt-bg.net/
Blog: http://live.gigavolt-bg.net/

13

I changed to one worker and the loads have gone down, thanks (spasibo!)
Igor

tho bandwidth usage is still 10% lower than lighttpd im gonna test on
other
servers now

i will back report results

14

CFQ - refinement, but powerful for extreme disk I/O

On Mon, Feb 25, 2008 at 4:01 AM, Dave C. [email protected] wrote:

My recommendation for IO bound workloads is to use the Deadline
Scheduler rather than the anticipatory or CFQ scheduler

On 25/02/2008, at 3:46 AM, Yordan G. wrote:

Use bonnie for test you storage space. And exucate “dmesg” (for
Linux OS) and view any network problem.

–
Please excuse my bad english…

С уважение,
Й. Георгиев.

WEB: http://gigavolt-bg.net/
Blog: http://live.gigavolt-bg.net/

15

On Mon, Feb 25, 2008 at 09:45:57AM +0000, nginx.mailinglist wrote:

I changed to one worker and the loads have gone down, thanks (spasibo!)
Igor

tho bandwidth usage is still 10% lower than lighttpd im gonna test on other
servers now

i will back report results

As I understand for writev-backend lighttpd mmap()s file in 512K chunks
and writev()s them. You may try in nginx

output_buffers 1 512k; # default is “1 32k”

The output_buffers are used if sendfile is not used.

Also you may try to set 2 or 3 workers if it will increase bandwidth.

Note, that disabling sendfile in both nginx and ligthy may increase
bandwidth, but also ceratinly increases memory consumption at user- and
kernel-level that may leave to DOS. You should find compromisse.

16

Increasing buffer size and number of workers from 1 only made things
worse,
at one stage the laod started spiraling above 20 its down to 4 now on
the 2
servers

Right now im very screwed, the performance loss from switching to nginx
is
killing the servers, i also setup 2nd server identically

the graphs speak for themselves :frowning:

server converted yesterday
http://img255.imageshack.us/img255/6753/31857516tg7.png

new server to the left of the dip is lighttpd
http://img341.imageshack.us/img341/3734/59174322lt9.png

im now very screwed as i was gonna relly on nginx’s accel-redirect
feature
and lighttpd 1.4.18 has it, im testing lighttpd1.5 svn now but i dont
know
how stable that be,
sorry for being so down :'{ i feel like crying this was certainly the
worst
case scenraio when is started migrating, a 10-20% drop in bandwdith
usage between all the servers will mean 1000-2000$ loss due to wasted
costs

17

On Mon, Feb 25, 2008 at 04:41:35PM +0000, nginx.mailinglist wrote:

server converted yesterday
case scenraio when is started migrating, a 10-20% drop in bandwdith
usage between all the servers will mean 1000-2000$ loss due to wasted costs

Are any crit or alert errors in error_log ?
“worker_connections 1024” may be low for load, it can be changed
to 5000 or 10000.

18

On 25.02.2008, at 17:41, nginx.mailinglist wrote:

server converted yesterday
the worst case scenraio when is started migrating, a 10-20% drop in
bandwdith usage between all the servers will mean 1000-2000$ loss
due to wasted costs

hey man…
why on earth are you changing your production server setup when you
know that you have troubles?

calm down, and start thinking… probably switch back to your old
setup and try the new setup on some development server first? did you
try the suggestions you received earlier?

sorry, but i don’t feel any sympathy with you.

jodok

ps.: we’re able to serve between 400 and 500 mbit with a single nginx-
server with no significant load. (but with enough memory for kernel
disk cache, large raid controller memory and really fast discs)

tho bandwidth usage is still 10% lower than lighttpd im gonna test

–
Igor S.
Igor Sysoev

–
“Beautiful is better than ugly.”
– The Zen of Python, by Tim Peters

Jodok B., Lovely Systems GmbH
Schmelzhütterstraße 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060

19

Hello,

You are top result is **http://pastebin.com/m2587b666 I view 70% wa!!!
This is problem! Server disk is very load up. Test disk ot RAID for seek
!

–
Please excuse my bad english…

С уважение,
Й. Георгиев.

WEB: http://gigavolt-bg.net/
Blog: http://live.gigavolt-bg.net/

20

Ok i got it stabilized at about 75mbit and loads of 2-6

im gonna have got to plan B get several more servers and put them on
internal lan run the php-cgi on them and connect to them using nginx
since
it connects over a socket (this should work easily right? im gonna setup
a
test setup and test this feature properly)

this way i be able to make most of the available and paid for bandwdith

the extra checks added by the php layer are creating more load im not
blaming nginx here, it is perfroming exceptionally, maybe my
expectations
were too high, the old lighttpd setup didnt touch php at all, but it
didnt
have much in way of functionality such as custom logging, download speed
control, authentication etc that are now possible thanks to nginx’s
resumable X-Accel-redirect feature

thanks everyone for the help