Enable reuseport then only one worker is working?

Server NGINX
1

Hi All,

I just upgrade Nginx from 1.8 o 1.9 on my FreeBSD box.
After I enabled reuseport on my server, it seems now there is one worker
always takes up 100% CPU and all the rest workers are use less than 1%
CPU.
In the day time it’s OK because my website doesn’t have lots of users.
But
at night it’s very slow.

Active connections: 5716
server accepts handled requests
175477 175477 1805645
Reading: 1 Writing: 698 Waiting: 5541

After I disable reuseport, it seems my website speed is back to normal.
Each
worker (18 workers in total) takes up about 10%-15% CPU.

I understand that enable reuseport will disable accept_mutex. But My
understanding is that Nginx will assign work load to different multiple
socket listeners then to different workers. BUt it seems in my
environment
all workload was assign to the same worker, even though it’s very busy
and
the rest are idle.

Did I miss anything in the configuration? or for a busy server, it’s
better
to use accept_mutex instead of reuseport?

Posted at Nginx Forum:

2

On Sunday 28 February 2016 08:52:12 meteor8488 wrote:

Hi All,

I just upgrade Nginx from 1.8 o 1.9 on my FreeBSD box.
[…]
Did I miss anything in the configuration? or for a busy server, it’s better
to use accept_mutex instead of reuseport?

[…]

In FreeBSD the SO_REUSEPORT option has completely different behavior
and shouldn’t be enabled in nginx.

wbr, Valentin V. Bartenev

3

Hello!

On Sun, Feb 28, 2016 at 08:52:12AM -0500, meteor8488 wrote:

I just upgrade Nginx from 1.8 o 1.9 on my FreeBSD box.
After I enabled reuseport on my server, it seems now there is one worker
always takes up 100% CPU and all the rest workers are use less than 1% CPU.
In the day time it’s OK because my website doesn’t have lots of users. But
at night it’s very slow.

While SO_REUSEPORT socket option is available on FreeBSD, its
behaviour is different from one nginx relies on: instead of
balancing between all sockets as Linux and DragonFly do, it
preserves historic behaviour for TCP and delivers all connections
to one socket instead.

That is, there is no surprise that you see all work done by a
single worker in your setup when you use “listen … reuseport” on
FreeBSD.

Note that documentation explicitly says it’s only expected to work
on Linux and DragonFly:

: This currently works only on Linux 3.9+ and DragonFly BSD.

See http//nginx.org/r/listen for details.

–
Maxim D.
http://nginx.org/

4

Do you know why FreeBSD does not do this? Is there a technical reason to
not
do that?

Posted at Nginx Forum:

5

Yes but is there a technical reason why it hasn’t been done yet? Does
FreeBSD have a reason to not do it? Just because Linux did does not mean
it
should be done.

Posted at Nginx Forum:

6

On 3/1/16 3:46 PM, drhowarddrfine wrote:

Do you know why FreeBSD does not do this? Is there a technical reason to not
do that?

Just because you need someone to write this code for FreeBSD.

–
Maxim K.

7

Hello,

On 2/28/16 11:22 PM, Валентин Бартенев wrote:

In FreeBSD the SO_REUSEPORT option has completely different behavior
and shouldn’t be enabled in nginx.

Should the configruation option then be disabled or silently ignored in
FreeBSD at this time?

–
Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

8

On 3/1/16 4:08 PM, drhowarddrfine wrote:

Yes but is there a technical reason why it hasn’t been done yet? Does
FreeBSD have a reason to not do it? Just because Linux did does not mean it
should be done.

Please approach FreeBSD developers.

–
Maxim K.

9

After a brief search, I was correct, and there are technical reasons for
not
doing this and Linux and Dragonfly are doing it wrong.

Posted at Nginx Forum:

10

Hi Jim,

On 01/03/16 13:10, Jim O. wrote:

to use accept_mutex instead of reuseport?

[…]

In FreeBSD the SO_REUSEPORT option has completely different behavior
and shouldn’t be enabled in nginx.

Should the configruation option then be disabled or silently ignored in
FreeBSD at this time?

It would be difficult to selectively ignore operating systems based on
how this function is supported. Especially if that support changes over
time.

I believe that DragonFly BSD for example supported the method most BSDs
use originally.

In the FreeBSD pattern it is designed to bleed off connections to one
service as another comes up. Such as a binary upgrade. The last socket
listener always gets new connections. This is almost the exact opposite
of what Linux does with the option (DragonFly is somewhere in the
middle). I’m personally not a BSD expert so can’t say which other BSD
operating systems use the same method as FreeBSD and even which ones do
not even have a SO_REUSEPORT option.

Documentation is probably the best option for now and we have tried to
make it clear which operating systems do support this feature.

Kind Regards

Andrew H. (LinuxJedi)
Technical Product Manager, NGINX Inc.

11

On Tuesday 01 March 2016 08:32:07 drhowarddrfine wrote:

After a brief search, I was correct, and there are technical reasons for not
doing this and Linux and Dragonfly are doing it wrong.

[…]

What is the reason?

wbr, Valentin V. Bartenev

12

Hello,

On 3/1/16 8:34 AM, Andrew H. wrote:

[…]
Should the configruation option then be disabled or silently ignored in
FreeBSD at this time?

It would be difficult to selectively ignore operating systems based on
how this function is supported. Especially if that support changes over
time.

I don’t claim to know how “difficult” that would be, but with all the
extremely talented coders in the Nginx group, I would think that
“difficult” would not be a barrier to “doing it right”. If OS support
changes, nginx can change. Something tells me that with a FreeBSD Core
Team member on the Nginx payroll, if this OS feature changes, it’ll
filter through to the people who write the code.

If I choose to try “use kqueue” on a system that does not support that
event method, I am hoping nginx would either tell me by refusing to
start, or ignore my stupidity. Similarly, if I attempt to enable
“reuseport” on Solaris or Windows which (I guess) do not support
SO_REUSEPORT, nginx will inform me. I know this happens on FreeBSD with,
for instance, aio. If aio is not built in to the kernel (it is not by
default), or specifically loaded, nginx gives an error and won’t start.
In the case of SO_REUSEPORT, the OS call has virtually the opposite of
the desired effect.

If a directive has the potential for significantly bad consequences, it
should be spat out during config testing with at least a warning.

not even have a SO_REUSEPORT option.

Documentation is probably the best option for now and we have tried to
make it clear which operating systems do support this feature.

I’m not trying to make an excuse for not reading the documentation,
which is clear and is exactly why I have not tested this feature on
FreeBSD. Rather, I’m suggesting a more rigorous approach to config
parsing.

–
Jim O.

“Never argue with a fool, onlookers may not be able to tell the
difference.” - Mark Twain

13

On 3/1/16 5:23 PM, Jim O. wrote:

Hi All,
In FreeBSD the SO_REUSEPORT option has completely different
how this function is supported. Especially if that support changes
over
time.

I don’t claim to know how “difficult” that would be, but with all
the extremely talented coders in the Nginx group, I would think that
“difficult” would not be a barrier to “doing it right”. If OS
support changes, nginx can change. Something tells me that with a
FreeBSD Core Team member on the Nginx payroll, if this OS feature
changes, it’ll filter through to the people who write the code.

Jim, we don’t have any FreeBSD core team members on payroll.

–
Maxim K.

14

On 01/03/16 14:23, Jim O. wrote:

Hi All,
and shouldn’t be enabled in nginx.
extremely talented coders in the Nginx group, I would think that
“difficult” would not be a barrier to “doing it right”. If OS support
changes, nginx can change. Something tells me that with a FreeBSD Core
Team member on the Nginx payroll, if this OS feature changes, it’ll
filter through to the people who write the code.

This would be more a case of effort / payoff. I’m not saying it isn’t
technically possible. But effort may be better spent implementing new
features people have been asking for (we have some kick-ass stuff
upcoming). Rather than implementing an OS kernel version detection with
blacklist.

If I choose to try “use kqueue” on a system that does not support that
event method, I am hoping nginx would either tell me by refusing to
start, or ignore my stupidity. Similarly, if I attempt to enable
“reuseport” on Solaris or Windows which (I guess) do not support
SO_REUSEPORT, nginx will inform me. I know this happens on FreeBSD with,
for instance, aio. If aio is not built in to the kernel (it is not by
default), or specifically loaded, nginx gives an error and won’t start.
In the case of SO_REUSEPORT, the OS call has virtually the opposite of
the desired effect.

Comparing it to kqueue isn’t quite the same. On operating systems that
do not support SO_REUSEPORT (including older Linux kernels) you will not
be able to use the option at all. The problem here is FreeBSD has the
option, it just has a different behaviour. I’m not even sure it is
possible to detect in a build system without maintaining a blacklist.

If a directive has the potential for significantly bad consequences, it
should be spat out during config testing with at least a warning.

I have mixed feelings on this. I believe for a majority of users this
isn’t a problem so users will see a warning that could concern them when
in reality in a majority of cases it doesn’t affect them.

Since it isn’t enabled by default a user would have to research and make
a conscious choice to turn it on. I would hope at this point the user
would have made an informed decision when enabling tuning directives.

Kind Regards

Andrew H. (LinuxJedi)
Technical Product Manager, NGINX Inc.

15

On 3/1/16 8:19 PM, Jim O. wrote:

it’s
Should the configruation option then be disabled or silently
the extremely talented coders in the Nginx group, I would think that

That, and the recent work sponsored by Nginx on FreeBSD sendfile(2)
to be included in the upcoming release (11). If he is no longer on
the payroll he is still working closely with you, so this hardly
invalidates my premise that you would be aware of future changes in
FreeBSD behavior. :wink:

Jim, that was 1.6+ years ago, this is not true anymore.

Can we return to the technical part of the discussion?

–
Maxim K.

16

Hello,

[…]
based on
Jim, we don’t have any FreeBSD core team members on payroll.
Perhaps you can understand my confusion when I see multiple references
to it online, including this tweet.
https://mobile.twitter.com/maximkonovalov/status/486847353484480512.

That, and the recent work sponsored by Nginx on FreeBSD sendfile(2) to
be included in the upcoming release (11). If he is no longer on the
payroll he is still working closely with you, so this hardly invalidates
my premise that you would be aware of future changes in FreeBSD
behavior. :wink:

Jim

17

On 3/1/16 8:50 PM, Jim O. wrote:

Hello,

[…]

nginx.

Should the configruation option then be disabled or
silently ignored in FreeBSD at this time?

[…]

My point is that I would like to see more rigorous, bullet-proof
config parsing/testing on the part of nginx. This is one example.
We can agree to disagree on its importance to users.

This is definitely something that makes perfect sense. We will try
to figure out how to fix it when time permits.

–
Maxim K.

18

Hi Guys,

Thanks for all these information.
But, is there any way for FreeBSD to enable it?

Posted at Nginx Forum:

19

Hello,

I just upgrade Nginx from 1.8 o 1.9 on my FreeBSD box.

I don’t claim to know how “difficult” that would be, but with all

That, and the recent work sponsored by Nginx on FreeBSD sendfile(2)
to be included in the upcoming release (11). If he is no longer on
the payroll he is still working closely with you, so this hardly
invalidates my premise that you would be aware of future changes in
FreeBSD behavior. :wink:
Jim, that was 1.6+ years ago, this is not true anymore.

Can we return to the technical part of the discussion?

Perhaps you missed the little wink there. My apologies.

My point is that I would like to see more rigorous, bullet-proof config
parsing/testing on the part of nginx. This is one example. We can agree
to disagree on its importance to users.

Jim

20

Hi,

Unfortunately no, the FreeBSD kernel doesn’t have the feature
implemented in the way you would want to use it.

Kind Regards
Andrew

On 02/03/16 00:13, meteor8488 wrote:

nginx Info Page

–
Andrew H. (LinuxJedi)
Technical Product Manager, NGINX Inc.