Too many open files

Ruby
1

I am writing an application server using TCP server which can only
accept upto 249 client connections. After that, I get ‘Error: Too many
open files…’. I run my server in Windows XP.

Windows XP has a limit of 255 files opened by any process, including
stdin, stdout and stderr. I googled for how to increase the limit, and
have not found a solution other than:

  1. modify main.c and call _setmaxstdio(n), which need recompile ruby;
  2. use Win32 API to accept connection.

I noticed that there are so many people post this problem but their
limits are different, some people have 512, some have 1024. looks like
this number is configurable. my machine is set to 255 which is way too
low. I can run multiple processes to get around this problem, but not 23
process to support 5000 connections.

Anything I can do to fix this? Thanks in ahead.

Qi

2

2009/5/7 Qi Lu [email protected]:

I noticed that there are so many people post this problem but their
limits are different, some people have 512, some have 1024. looks like
this number is configurable. my machine is set to 255 which is way too
low. I can run multiple processes to get around this problem, but not 23
process to support 5000 connections.

Anything I can do to fix this? Thanks in ahead.

AFAIK you need to get yourself a copy of a Server version of Windows
because the limit is built into the OS version (or license).
Microsoft’s reasoning is that if you plan to put up a server which can
sustain higher loads then you are probably doing this for commercial
reasons and hence should pay for it. Apart from that you will also
get more tools do administer the server.

If you do not have a fixed platform requirement, then getting a Linux
box is probably the cheaper choice.

Kind regards

robert

3

Qi Lu wrote:
I googled for how to increase the limit, and

have not found a solution other than:

  1. modify main.c and call _setmaxstdio(n), which need recompile ruby;
  2. use Win32 API to accept connection.

What about running a cygwin version of Ruby?

4

2009/5/8 Albert S. [email protected]:

Qi Lu wrote:
I googled for how to increase the limit, and

have not found a solution other than:

  1. modify main.c and call _setmaxstdio(n), which need recompile ruby;
  2. use Win32 API to accept connection.

What about running a cygwin version of Ruby?

Won’t help if he is hitting the OS limit.

robert

5

Windows XP has a limit of 255 files opened by any process, including
stdin, stdout and stderr. I googled for how to increase the limit, and
have not found a solution other than:

A few things since I ran into this problem before:
msvcrt 6 (ruby’s default) has a limit of 512. Ruby itself “duplicates”
fd objects when created, so that it can have the “original” and a more
“linuxy” equivalent (I think that’s why–it explains it more in the
source I just glanced at it once). So with msvcrt 6 and ruby you’re
limited at 512.

That being said, be very aware that select calls are limited to 64
sockets MAX. So even though you can open 256, don’t use more than 64
ever ever ever. Though I suppose now that we have Ruby 1.9 (have as
in…it compiles [1]) you could use multiple threads, and, per thread
pass in 64. I think that would get around that one limitation–I think
:slight_smile:
Actually it might be possible to through them all into a single select
(single threaded), thanks to Usaku Nakamura’s fixing this bug [1].

For 1.8.x, to get around the select limitation you’d need to set
FD_SETSIZE to at least 512, then recompile ruby [2].

So msvcrt 6’s limit is 512. I think you could use a newer version of
msvcrt (i.e…compile ruby with VS2008 or what not) and that would give
you up to windows’ default max–which is either 1024 or 2048, unless you
have the server version of windows, which is 1000’s.
At least that’s my take on the problem I haven’t looked into it too
much.

The “real” fix to this would probably be (if you’re opening sockets) to
write a C extension that works around it somehow
So anyway the only way to really get around this problem is to do your
#2 (use win32 api to open files, instead of just fopen). Maybe #1 would
help, slightly, I haven’t tried that out, either.

If you’re looking to recompile ruby (using mingw),
GitHub - oneclick/rubyinstaller: RubyInstaller for Windows - Build recipes has an easy way to
do so.

GL.
-r

  1. modify main.c and call _setmaxstdio(n), which need recompile ruby;
  2. use Win32 API to accept connection.

I noticed that there are so many people post this problem but their
limits are different, some people have 512, some have 1024. looks like
this number is configurable. my machine is set to 255 which is way too
low. I can run multiple processes to get around this problem, but not 23
process to support 5000 connections.

Anything I can do to fix this? Thanks in ahead.

Qi

[1] http://redmine.ruby-lang.org/issues/show/755 – 1.9 only,
unfortunately.
[2] http://redmine.ruby-lang.org/issues/show/670

6

Thank you very much for your reply. It makes me understand the limit.
But why do I have only 256 limit not even 512? I use Ruby 1.8.6 version
and run on XP. If I can make my limit to 512, it will help little bit.

It’s 256 because after ruby code opens any file descriptor it “dup’s”
them for internal use (for some reason–I can’t remember why it’s
explained in the code).

Since my application is not that time critical, I use a single thead to
do the select. and slice all sockets into multiple a group of 64 i/o
array. I have got around of 64 limit. my problem is realy 256 max open
file limit. What I can do to make it at least 512?

Maybe by recompiling ruby using a newer MSVC compiler. Or figuring out
how to compile the mingw one and have it link against a different
version of msvcrt (I’ve heard it’s possible to have mingw link against
an arbitrary version of msvcrt.dll–it defaults to version 6–I have not
yet heard of anybody trying that with ruby, though it should be
possible).

You might be able to use rev instead (which which require a rewrite, but
might give you more descriptors)–I’ll check on its limit sometime.

http://github.com/rogerdpack/rev/tree/master
version
compiles well for mingw.
-=r

7

Roger P. wrote:

Windows XP has a limit of 255 files opened by any process, including
stdin, stdout and stderr. I googled for how to increase the limit, and
have not found a solution other than:

A few things since I ran into this problem before:
msvcrt 6 (ruby’s default) has a limit of 512. Ruby itself “duplicates”
fd objects when created, so that it can have the “original” and a more
“linuxy” equivalent (I think that’s why–it explains it more in the
source I just glanced at it once). So with msvcrt 6 and ruby you’re
limited at 512.

Thank you very much for your reply. It makes me understand the limit.
But why do I have only 256 limit not even 512? I use Ruby 1.8.6 version
and run on XP. If I can make my limit to 512, it will help little bit.

That being said, be very aware that select calls are limited to 64
sockets MAX. So even though you can open 256, don’t use more than 64
ever ever ever. Though I suppose now that we have Ruby 1.9 (have as
in…it compiles [1]) you could use multiple threads, and, per thread
pass in 64. I think that would get around that one limitation–I think
:slight_smile:
Actually it might be possible to through them all into a single select
(single threaded), thanks to Usaku Nakamura’s fixing this bug [1].

Since my application is not that time critical, I use a single thead to
do the select. and slice all sockets into multiple a group of 64 i/o
array. I have got around of 64 limit. my problem is realy 256 max open
file limit. What I can do to make it at least 512?

Thanks

Qi