Nginx flv stream gets too slow on 2000 concurrent connections

And also the 20+ lines of vmstat are given below with 2.6.32 kernal :-

procs -----------memory---------- —swap-- -----io---- --system–
-----cpu-----
r b swpd free buff cache si so bi bo in cs us sy
id
wa st
0 1 0 259020 49356 31418328 0 0 64 24 0 4 5
0
95 0 0
1 0 0 248100 49356 31418564 0 0 704 4 35809 3159 0
1
99 0 0
0 0 0 245248 49364 31419856 0 0 1340 48 35114 3217 0
0
99 0 0
1 0 0 243884 49364 31421084 0 0 940 4 35176 3106 0
0
99 0 0
0 0 0 243512 49364 31422152 0 0 812 4 35837 3204 0
0
99 0 0
0 0 0 241608 49364 31423056 0 0 1304 4 35585 3177 1
1
98 0 0
1 0 0 241076 49364 31424132 0 0 1004 4 35774 3199 0
0
99 0 0
0 0 0 241332 49372 31424644 0 0 724 76 35526 3203 0
0
99 0 0
0 0 0 240464 49372 31425376 0 0 776 4 35968 3162 0
0
99 0 0
0 1 0 238236 49372 31426244 0 0 652 4 35705 3131 0
0
99 0 0
0 0 0 234632 49372 31426924 0 0 1088 4 36220 3309 0
1
99 0 0
0 0 0 233640 49372 31428492 0 0 872 4 35663 3235 0
1
99 0 0
0 0 0 232896 49376 31429016 0 0 1272 44 35403 3179 0
0
99 0 0
1 0 0 231024 49376 31430064 0 0 528 4 34713 3238 0
0
99 0 0
0 0 0 239644 49376 31430564 0 0 808 4 35493 3143 0
1
99 0 0
3 0 0 241704 49376 31431372 0 0 612 4 35610 3400 1
1
97 0 0
1 0 0 244092 49376 31432028 0 0 280 4 35787 3333 1
1
99 0 0
2 0 0 244348 49376 31433232 0 0 1260 8 34700 3072 0
0
99 0 0
0 0 0 243908 49384 31433728 0 0 512 32 35019 3145 0
1
99 0 0
1 0 0 241104 49384 31435004 0 0 1440 4 35586 3211 0
1
99 0 0
0 0 0 234600 49384 31435476 0 0 868 4 35240 3235 0
1
99 0 0
procs -----------memory---------- —swap-- -----io---- --system–
-----cpu-----
r b swpd free buff cache si so bi bo in cs us sy
id
wa st
1 0 0 233656 49384 31436376 0 0 704 4 35297 3126 0
1
99 0 0
0 0 0 233284 49384 31437176 0 0 192 4 35022 3202 0
0
99 0 0
0 0 0 228952 49392 31437336 0 0 868 32 34986 3211 0
1
99 0 0
0 0 0 232176 49392 31438124 0 0 448 4 35785 3294 0
1
99 0 0
0 0 0 230076 49392 31438664 0 0 1052 4 35532 3297 1
1
98 0 0
1 0 0 231184 49392 31439608 0 0 436 4 34967 3177 0
1
99 0 0
1 0 0 224300 49392 31440044 0 0 624 4 34577 3216 0
1
99 0 0
0 0 0 223748 49396 31440664 0 0 460 44 34415 3155 0
0
99 0 0
1 0 0 223260 49396 31441612 0 0 768 4 35287 3194 0
1
99 0 0
0 0 0 230464 49396 31441996 0 0 772 4 35140 3208 0
0
99 0 0
1 0 0 225504 49396 31442668 0 0 564 4 35316 3133 0
0
99 0 0

On Thu, Jan 24, 2013 at 12:00 AM, shahzaib shahzaib

I am seeing the following messages on dmesg output :-

TCP: Peer 79.211.64.145:54649/80 unexpectedly shrunk window
347253187:347272955 (repaired)
TCP: Peer 79.211.64.145:54649/80 unexpectedly shrunk window
347253187:347272955 (repaired)
TCP: Peer 79.211.64.145:54649/80 unexpectedly shrunk window
347253187:347272955 (repaired)
TCP: Peer 81.155.221.33:53075/80 unexpectedly shrunk window
1986341072:1986342532 (repaired)
TCP: Peer 81.155.221.33:53075/80 unexpectedly shrunk window
1986341072:1986342532 (repaired)
TCP: Peer 81.155.221.33:53075/80 unexpectedly shrunk window
1986341072:1986342532 (repaired)
TCP: Peer 79.211.64.145:54709/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)
TCP: Peer 79.211.64.145:54709/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)
TCP: Peer 79.211.64.145:54709/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)
TCP: Peer 79.211.64.145:54709/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)
TCP: Peer 79.211.64.145:54709/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)
TCP: Peer 79.211.64.145:54709/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)

If somebody can explain me ? what is that shrunk window thing?

<And if someone can get me that chinese blog for setting 2k concurrent
connections using sysctl-settings.>

Was it this one? It refers to 2M connections and claimed success.
http://rdc.taobao.com/blog/cs/?p=1062
The blog is in Chinese. I used Chrome and clicked on “Translate” to read
it.

Posted at Nginx Forum:

Am 23.01.2013 um 22:50 schrieb “richardm” [email protected]:

<And if someone can get me that chinese blog for setting 2k concurrent
connections using sysctl-settings.>

Was it this one? It refers to 2M connections and claimed success.
http://rdc.taobao.com/blog/cs/?p=1062

Yes, that’s it.

2000k, not 2k.

On 23 January 2013 20:42, Geoffrey H. [email protected]
wrote:

Just to say… You are already hitting 1Gbits with a 1Gbit port…
It’s normal that is slow… no?

+1. I’m not seeing the problem here.

Jonathan

Just to say… You are already hitting 1Gbits with a 1Gbit port…
It’s normal that is slow… no?

2013/1/23 shahzaib shahzaib [email protected]:

TCP: Peer 81.155.221.33:53075/80 unexpectedly shrunk window
1128744179:1128773611 (repaired)
wrote:

The load(nload) of 1500+ concurrent connections with 1Gbps port is :
What should i see into dmesg to analyse the problem ? I’ll also send you
TiB)

[email protected]
nginx Info Page


nginx mailing list
[email protected]
nginx Info Page


Geoffrey HARTZ

On Jan 23, 2013, at 5:04 PM, Lukas T. wrote:

I suggest to monitor your eth0 links carefully and upgrade to either multiple
bonded 1Gig links, 10gig links or more servers.

Or do what the cable companies do: compress your video until it fits,
regardless of how bad the quality gets :wink:


Scott R.
[email protected]
http://www.elevated-dev.com/
(303) 722-0567 voice

The load(nload) of 1500+ concurrent connections with 1Gbps port is : Curr:
988.95 MBit/s

Just to say… You are already hitting 1Gbits with a 1Gbit port…
It’s normal that is slow… no?
+1. I’m not seeing the problem here.

Exactly, the issue is crystal clear. You are already hitting your max
bandwidth with 1500+ concurrent connections, of course with 2000+
concurrent connections users will notice severe slowdowns. You have not
enough bandwidth to serve your clients.

I suggest to monitor your eth0 links carefully and upgrade to either
multiple bonded 1Gig links, 10gig links or more servers.

On Wed, Jan 23, 2013 at 7:04 PM, Lukas T. [email protected]
wrote:

The load(nload) of 1500+ concurrent connections with 1Gbps port is : Curr:
988.95 MBit/s
Just to say… You are already hitting 1Gbits with a 1Gbit port…
It’s normal that is slow… no?
+1. I’m not seeing the problem here.

Exactly, the issue is crystal clear. You are already hitting your max bandwidth
with 1500+ concurrent connections, of course with 2000+ concurrent connections
users will notice severe slowdowns. You have not enough bandwidth to serve your
clients.

I suggest to monitor your eth0 links carefully and upgrade to either multiple
bonded 1Gig links, 10gig links or more servers.

bah, you need a CDN.

Thanks for helping me out guyz. I’ll tune my content server according to
that chinese guide. Please keep in mind i had only sent the output of
one
of Five content servers. Other servers load(nload) is not that high and
they just hit 500Mbit/s on 2000 concurrent connections. However i’ll
monitor eth0 port more closely on peak time and will let you know the
status. Thanks :slight_smile:

I have got an idea of preventing users to download videos from our site,
so
they just can stream videos and that way will save our bandwidth. We
have
used one of nginx module “limit_conn 1” so nobody will be able to
download
stream. But this thing has a major drawback of stream i.e if four users
are
streaming videos under a LAN network with same ip, other 3 won’t be able
to
stream videos due to 1st user, who’s already streaming it and when he’ll
finish streaming it’ll resume for 2nd user and vice virsa.

Can someone guide me if we can just prevent downloading but stream
remains
the same ?

On Thu, Jan 24, 2013 at 11:57 AM, shahzaib shahzaib

Maybe not the box hits the 1Gbit band bandwidth, your switch also
possible
hits the limit.

2013/1/24 shahzaib shahzaib [email protected]