How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there’s nothing yet, just return.
Failing that, is there a way to manually check if there’s anything to
read or the read will block?
(Note that I’m developing for both 'nix and Windows.)
On Wed, 1 Nov 2006 [email protected] wrote:
How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there’s nothing yet, just return.Failing that, is there a way to manually check if there’s anything to
read or the read will block?(Note that I’m developing for both 'nix and Windows.)
it can’t be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if
there is
no input?
regards.
-a
On Oct 31, 2006, at 6:23 PM, [email protected] wrote:
On Wed, 1 Nov 2006 [email protected] wrote:
How can I perform a nonblocking IO read? That is, read whatever
is there to read and return. If there’s nothing yet, just return.
That’s what the Unix read() function does (“man 2 read”), available
in Ruby as IO#sysread(). The ri documentation doesn’t explain much
about how to use Ruby’s sysread, unfortunately.
Failing that, is there a way to manually check if there’s anything
to read or the read will block?(Note that I’m developing for both 'nix and Windows.)
The select() function (Kernel#select, in Ruby) is the standard way to
do IO multiplexing under Unix. You give it a list of file handles on
which you’re waiting to be able to read or write, and it returns when
one or more of them is ready. Windows provides a select() function,
too, but it only supports sockets (not pipes or ordinary file IO.)
The Win32 API generally doesn’t provide non-blocking IO methods,
because it’s assumed you’ll use threads if you want to do something
else while waiting for IO.
it can’t be done. search the archives, this sort of thing almost
always
indicates a design flaw. for instance - what will your program do
if there is
no input?
Check other IO channels, wait for other sorts of events, do work -
whatever. Where do come from asserting that the use of non-blocking
IO represents a “design flaw”?
Tom
In the interests of promoting creativity, I’ll ask the question without
any prior assumptions:
I’m using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I don’t
want to hang. Just keep on moving. In fact, stderr will normally be
empty.
How could I do this?
(I’m supporting 'nix and Windows)
[email protected] wrote:
it can’t be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?
I’m running an external command (using popen4 - yes, it is available on
Windows also). I capture stdout and stderr. “what will your program do
if there is no input?” - I assume you mean output - Well, I hope there
is no stderr output. If there is, I’d like to capture it and feed it
to Logger.
Likewise, if there is a problem, and there’s no stdout, then I don’t
want to wait for ever - I want to throw an exception.
On Wed, 1 Nov 2006 [email protected] wrote:
I’m running an external command (using popen4 - yes, it is available on
Windows also). I capture stdout and stderr. “what will your program do if
there is no input?” - I assume you mean output - Well, I hope there is no
stderr output. If there is, I’d like to capture it and feed it to Logger.Likewise, if there is a problem, and there’s no stdout, then I don’t want to
wait for ever - I want to throw an exception.
so, basically you want to timeout on a read right? in unix you can just
use
require ‘timeout’
Timeout::timeout(n) do
Open4.popen4 …
…
…
end
but, this is implimented via a thread so i think the read is going to
block
your entire process in windows… you might try it.
-a
On Wed, 1 Nov 2006, Tom P. wrote:
On Oct 31, 2006, at 6:23 PM, [email protected] wrote:
On Wed, 1 Nov 2006 [email protected] wrote:
How can I perform a nonblocking IO read? That is, read whatever is there
to read and return. If there’s nothing yet, just return.That’s what the Unix read() function does (“man 2 read”), available in Ruby
as IO#sysread(). The ri documentation doesn’t explain much about how to use
Ruby’s sysread, unfortunately.
no, that blocks. try this:
harp:~ > ruby -e ‘STDIN.sysread(1)’
it blocks. the OP wants a non-blocking call. eg
harp:~ > ruby -r io/nonblock -e ‘STDIN.nonblock{ STDIN.sysread(1) }’
-e:1:in sysread': Resource temporarily unavailable (Errno::EAGAIN) from -e:1 from /home/ahoward//lib/ruby/1.8/io/nonblock.rb:19:innonblock’
from -e:1
The select() function (Kernel#select, in Ruby) is the standard way to do IO
multiplexing under Unix. You give it a list of file handles on which you’re
waiting to be able to read or write, and it returns when one or more of them
is ready. Windows provides a select() function, too, but it only supports
sockets (not pipes or ordinary file IO.) The Win32 API generally doesn’t
provide non-blocking IO methods, because it’s assumed you’ll use threads if
you want to do something else while waiting for IO.
right. but ruby’s thread are green and, on windows, block the entire
process
when waiting on IO so not an option here.
Check other IO channels, wait for other sorts of events, do work - whatever.
Where do come from asserting that the use of non-blocking IO represents a
“design flaw”?
because nine times out of ten code has nothing to do if no input is
available.
if it does that’s another story, but then there are basically two
options:
use select or an event abstraction in one giant loop. if you do this
you are
going to block anyhow, only based on all inputs, not just one.
use threads, only you can’t if you’re wanting ruby code to run nb in
windows.
i’m not saying there is never a case for nbio - i use it myself - just
that,
unless a developer has a good reason to be doing something else it’s not
required and, as a search of ruby-talk will show, that is often the
case.
regards.
-a
On Oct 31, 2006, at 5:30 PM, S. Robert J. wrote:
Not very attractive…
Back to the original question: there is no simple way to read
nonblocking, or see if any output is available ?!?!
in io/wait:
— IO#ready?
returns non-nil if input available without blocking, or nil.
also: ri IO.read_nonblock
[[email protected]]
|
| it can’t be done. search the archives, this sort of thing almost
| always indicates a design flaw. for instance - what will your
| program do if there is no input?
nonblocking IO is not a design-flaw; it is something that is in common
use and is supported by most languages with a decent socket API. on
some OS’es this is even available for file IO (although widespread
language support seems to lag behind).
Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 “New I/O APIs for the JavaTM Platform”).
for examples of use you might want to check out the Reactor pattern
and other patterns for concurrent programming.
-Bjørn
Not very attractive…
Back to the original question: there is no simple way to read
nonblocking, or see if any output is available ?!?!
On Oct 31, 2006, at 7:20 PM, [email protected] wrote:
How could I do this?
Have you tried simply reading from them? When the command
terminates, that should close those pipes on the command’s end. At
that point, a simple read on the stdout handle, for instance, will
return whatever output there was and then eof. If there’s no output,
the read should return immediately. Did you try that yet? Did you
try the (Windows-oriented) example from the ‘ri Open4’ page? If
you’re working in Unix you could try
status = POpen4::poen4(‘cat -’) do |stdout, stderr, stdin|
stdin.puts ‘hello world’
stdin.close
puts “stdout: #{stdout.read.strip}”
puts “stderr: #{stderr.read.strip}”
end
Anyway, you would only need nonblocking IO if you wanted to read bits
of the stderr stream before the command exited, but that doesn’t
sound like what you’re want.
Tom
On Wed, 1 Nov 2006, Tom P. wrote:
The select() function (Kernel#select, in Ruby) is the standard way to do IO
multiplexing under Unix.
At a very deep level (last time I looked) thread animation and IO met
and became the same thing in a giant select lurking in the core of the
ruby Interpreter.
ie. Use select / spin another thread amounted cosmically to the same
thing. Choose whatever is easiest for you.
John C. Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : [email protected]
New Zealand
“We have more to fear from
The Bungling of the Incompetent
Than from the Machinations of the Wicked.” (source unknown)
On Oct 31, 2006, at 9:37 PM, Tom P. wrote:
If you’re working in Unix you could try
status = POpen4::poen4(‘cat -’) do |stdout, stderr, stdin|
D’oh! (typo) That should have been
status = POpen4::popen4(‘cat -’) do |stdout, stderr, stdin|
stdin.puts ‘hello world’
stdin.close
puts “stdout: #{stdout.read.strip}”
puts “stderr: #{stderr.read.strip}”
end
Tom
[Tom P. [email protected]]
|
| The select() function (Kernel#select, in Ruby) is the standard way to
| do IO multiplexing under Unix. You give it a list of file handles on
| which you’re waiting to be able to read or write, and it returns when
| one or more of them is ready. Windows provides a select() function,
| too, but it only supports sockets (not pipes or ordinary file IO.)
if memory serves, a select(2) on a file descriptors representing a
(disk) file only had defined semantics when used to wait on mandatory
locking and is not used for ensuring that a read(2) or write(2) will
not block. (I might be in error since it has been a while since I
looked into it).
-Bjørn
On Wed, 1 Nov 2006, Bjorn B. wrote:
Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 “New I/O APIs for the JavaTM Platform”).
but it’s archaic compared to event driven frameworks like
libevent
http://liboop.ofb.net/
in addition it’s a hack in the c lib of many oses.
for examples of use you might want to check out the Reactor pattern
and other patterns for concurrent programming.
afaik the reactor pattern is a synchronous pattern
artima - Comparing Two High-Performance I/O Design Patterns
not unlike the model of libevent and liboop - which are both
synchronous…
am i missing something?
-a
On 10/31/06, [email protected] [email protected] wrote:
Are you having an interactive conversation with the process created by
popen4? Is that why you need nbio, because you need to poll more than
one
stream simultaneously?
You can get the file descriptors associated with stdout and stderr
(they’ll
be 1 and 2 unless you did something unusual), set them nonblocking, and
select them for reading with a short timeout.
On 10/31/06, [email protected] [email protected] wrote:
use and is supported by most languages with a decent socket API. on
some OS’es this is even available for file IO (although widespread
language support seems to lag behind).Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 “New I/O APIs for the JavaTM Platform”).but it’s archaic compared to event driven frameworks like
Ara, I’m curious to know why you think the usage of nbio is archaic or
indicative of a design flaw. I wrote the EventMachine library in order
to
make certain kinds of Ruby programs easier to write, and it uses nbio
pervasively. I’m fairly sure libevent does, also. EM works with almost
any
kind of descriptor, except for certain things on Windows (like _pipe()
descriptors, which are nonselectable, and some of the native Windows
objects
that don’t support a socket-like API).
Up until recently (late May 2006), Ruby didn’t have complete support for
nbio on all descriptor types, which is one of the reasons that
EventMachine
includes a compiled C++ extension. EM interoperates perfectly well with
Ruby
threads,* which is another important reason to use nbio.
*Except for certain cases on Windows where Ruby code that accesses
standard
descriptors blocks all other Ruby threads. But this is not an EM issue,
On 11/1/06, [email protected] [email protected] wrote:
ready so i… oops, now it is ready), and stdio buffers being lost.
- something else could simoultaneously be done, but the problem could
his insight is enough to raise ‘too low-level’ warnings in my mind! at
least
until using nbio from a ruby script doesn’t require such deep
understanding of
ruby’s internals
Thanks for clarifying, Ara. Having put a lot of work into domesticating
nbio
for Ruby programs, all I can say is I agree with you.
I still don’t feel like I understand what the OP really is trying to do.
Feels like it shouldn’t be such a hard problem. Maybe nbio is just a red
herring in this case (although it does appear in the first sentence of
the
OP).
On Wed, 1 Nov 2006, Francis C. wrote:
includes a compiled C++ extension. EM interoperates perfectly well with Ruby
threads,* which is another important reason to use nbio.
hi francis-
it’s exactly things like eventmachine that make me say using nbio is
archaic -
i don’t need to handle the complexities of nbio when powerful
abstractions
like it exist!
for instance, consider the OP’s orignial question: basically he wanted
to
timeout if stdout was not produced in a certain amount of time. i
certainly
wouldn’t delve into the complexities of nbio to handle this, but instaed
might
do something along the lines of
harrp:~ > cat a.rb
require ‘open4’ and require ‘timeout’
def might_take_too_long cmd, stdin, timeout = 42
stdout, stderr = ‘’, ‘’
Timeout::timeout(timeout) do
open4.popen4(cmd) do |cid,i,o,e|
i.write stdin and i.close
o.each{|line| stdout << line} and e.each{|line| stderr << line}
end
end
[ stdout, stderr ]
end
p might_take_too_long(‘ruby’, ‘sleep 1 and p 42’, 2)
p might_take_too_long(‘ruby’, ‘sleep 42 and p 42’, 2)
harp:~ > ruby a.rb
[“42\n”, “”]
/home/ahoward//lib/ruby/1.8/timeout.rb:43:in `might_take_too_long’:
execution expired (Timeout::Error)
now, timeout might use any manner of select and/or nbio - i don’t care.
my
point is just that, for 90-99% of the problems seem to want to solve
with
nbio, it is way too low level a mechanism to skin the cat and prefer to
stand
on the shoulders of powerful abstractions, like eventmachine or threads
or
whatever, and avoid the painful details of EAGAIN, race conditions (it
wasn’t
ready so i… oops, now it is ready), and stdio buffers being lost.
for the record, i have code which uses nbio, io/nonblock, etc. it’s
simply my
observation that a majority of the posts to this list about nbio have
two
issues:
nothing else can really be done while waiting for io. eg. nbio
isn’t
needed for the problem at hand
something else could simoultaneously be done, but the problem could
be
more elegantly solved with threads, queues, readpartial, or
something even
more abstract
the rest, of course, really do need nbio. a quick search of archives
though
is an eye opener into the difficulties of mixing nbio and stdio, check
out
these threads, specifically the posts by tanaka akira, who knows about
10
billion times more about nbio and ruby than i do
his insight is enough to raise ‘too low-level’ warnings in my mind! at
least
until using nbio from a ruby script doesn’t require such deep
understanding of
ruby’s internals
kind regards.
-a
Tom P. wrote:
Anyway, you would only need nonblocking IO if you wanted to read bits of
the stderr stream before the command exited, but that doesn’t sound like
what you’re want.
Actually this is not correct: if there is a lot written to stderr then
you need to read that concurrently. If you do not do that then the
process will block on some stderr write operation that fills up the pipe
and you get a deadlock because your code waits for process termination.
Kind regards
robertThis forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs