"Dummy" IO object to push and pull data?

shayhok · January 4, 2010, 5:41pm

Test program:

Required support, go to process dir

require(“open3.so”)
Dir.chdir("…")

Initiate the process and its streams

command = “********”
inline, outline, errline = Open3.popen3(command)

Quick function for short timestamps

require(“time”)
def tstamp(line)
puts("#{Time.now().to_s().split(" “)[3]} #{line}”)
end

Read from stderr, thread stdin input

loop do
tstamp("> Awaiting data…")
line = errline.gets().strip()
tstamp(“Output: #{line}”)

if(line == “0”)
tstamp("> Received the required line.")

Thread.new() do
  tstamp("> The thread has been created.")
  sleep(5)

  tstamp("> Time's up! Sending input...")
  inline.puts("EXIT")
  tstamp("> The input has been sent.")
end

end
end

Output:

11:35:03 > Awaiting data…
11:35:03 Output: 2
11:35:03 > Awaiting data…
11:35:13 Output: 3
11:35:13 > Awaiting data…
11:35:23 Output: 0
11:35:23 > Received the required line.
11:35:23 > The thread has been created.
11:35:23 > Awaiting data…
11:35:33 Output: 4
11:35:33 > Awaiting data…
11:35:43 Output: 1
11:35:43 > Awaiting data…

shayhok · January 4, 2010, 8:50pm

On 1/4/10, Shay H. [email protected] wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

Ah, that’s right. On windows, there’s a special method you have to use
instead of select to check if input is available on a pipe. Let’s see
… I believe the method you’re looking for is kbhit (or _kbhit
maybe? ms likes their underscores…) Check this page on MSDN:
http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx

There should be some library somewhere that makes this available to
you in rubyland… Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.

shayhok · January 4, 2010, 2:41pm

Shay H. wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

Hm, let me give an example of what this would output. The process that
I’m creating the stream for outputs something every 10 seconds. As I
mentioned, if it outputs a 0, I want to give it an input of “Received 0
five seconds ago.” five seconds later.

0:00:00 - Program started.
0:00:10 - Output: 3
0:00:20 - Output: 0
0:00:30 - Output: 2
0:00:30 - Input: Received 0 five seconds ago.

It’s a rough example, yes, but the point is that stdin seems to wait
until stderr is free. I understand this may be normal, but is there an
alternative to select() that will prevent me from tying up the streams
if there’s no ready data in stderr?

shayhok · January 4, 2010, 9:23pm

On 1/4/10, Caleb C. [email protected] wrote:

… I believe the method you’re looking for is kbhit (or _kbhit
maybe? ms likes their underscores…) Check this page on MSDN:
http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx

There should be some library somewhere that makes this available to
you in rubyland… Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.

Except when I actually read the msdn link above, I see it only works
with stdin… that may not help you. I think there might be some other
windows-specific call you can make that checks any fd for input
availability… you might have to hunt around to find it, or ask on a
windows programming list.

shayhok · January 4, 2010, 9:07pm

Shay H. wrote:

Test program:
…
tstamp("> Time’s up! Sending input…")

But that message never appears in the output you showed - and this line
occurs before the puts. So at the moment looks either (a) sleep(5) is
sleeping indefinitely, or (b) the whole thread has terminated with an
exception.

A better test would be a standalone pair of ruby programs, because then
you have something which anyone can reproduce the problem with (*).

Here’s your test program, very slightly modified, with a partner
program.

==> prg1.rb <==
Thread.abort_on_exception = true
require “open3”

command = “ruby prg2.rb”
inline, outline, errline = Open3.popen3(command)

def tstamp(line)
puts("#{Time.now().to_s().split(" “)[3]} #{line}”)
end

Read from stderr, thread stdin input

loop do
tstamp("> Awaiting data…")
line = errline.gets().strip()
tstamp(“Output: #{line}”)

if(line == “0”)
tstamp("> Received the required line.")

Thread.new() do
  tstamp("> The thread has been created.")
  sleep(5)

  tstamp("> Time's up! Sending input...")
  inline.puts("EXIT")
  tstamp("> The input has been sent.")
end

end
end

==> prg2.rb <==
i = -2
loop do
$stderr.puts i
i += 1
if select([$stdin],nil,nil,2)
line = $stdin.gets.chomp
$stderr.puts “>>> Received #{line} <<<”
end
end

And if I run it under Linux, here’s what I get:

$ ruby prg1.rb
19:58:05 > Awaiting data…
19:58:05 Output: -2
19:58:05 > Awaiting data…
19:58:07 Output: -1
19:58:07 > Awaiting data…
19:58:09 Output: 0
19:58:09 > Received the required line.
19:58:09 > The thread has been created.
19:58:09 > Awaiting data…
19:58:11 Output: 1
19:58:11 > Awaiting data…
19:58:13 Output: 2
19:58:13 > Awaiting data…
19:58:14 > Time’s up! Sending input…
19:58:14 > The input has been sent.
19:58:14 Output: >>> Received EXIT <<<
19:58:14 > Awaiting data…
19:58:14 Output: 3
19:58:14 > Awaiting data…
19:58:16 Output: 4
19:58:16 > Awaiting data…
19:58:18 Output: 5
19:58:18 > Awaiting data…
^Cprg1.rb:13:in gets': Interrupt from prg1.rb:13 from prg1.rb:11:inloop’
from prg1.rb:11

What happens if you run this under Windows?

Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep and receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do “ruby prg2.rb” by itself. It
should print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received … <<<

What I’m trying to say is, if you can make prg2.rb work properly (or
make another of version of prg2 work using threads), then you ought to
be able to make prg1.rb work properly too.

(*) Incidentally: for someone to reproduce this problem, they’ll need to
know exactly what platform you have. You said “Windows Vista” but you
didn’t say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)

Regards,

Brian.

shayhok · January 4, 2010, 11:23pm

Shay H. wrote:

C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!

Received Hello World! <<<
-1
C:\Users\Shay\Desktop>Blah~

Received Blah~ <<<
0

And so forth and so on.

It returns you to a command prompt? How odd.

What about a threaded version of prg2.rb, does it behave any
differently?

Thread.new do
while line = $stdin.gets
$stderr.puts “>>> Received #{line.chomp} <<<”
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

shayhok · January 4, 2010, 11:43pm

Brian C. wrote:

It returns you to a command prompt? How odd.

What about a threaded version of prg2.rb, does it behave any
differently?

Thread.new do
while line = $stdin.gets
$stderr.puts “>>> Received #{line.chomp} <<<”
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

Oh, my apologies, I had put the user info there to show that I was
giving input (as if @echo hadn’t been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.

shayhok · January 4, 2010, 10:00pm

Brian C. wrote:

What happens if you run this under Windows?

C:\Users\Shay\Desktop>ruby prg1.rb
15:51:20 > Awaiting data…
15:51:21 Output: -2
15:51:21 > Awaiting data…

Then nothing else happens. I’m 90% sure this is because Windows, being
the operating system that it is, holds up everything when .gets() is
called on stdin until something is provided.

Brian C. wrote:

Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep and receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do “ruby prg2.rb” by itself. It
should print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received … <<<

C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!

Received Hello World! <<<
-1
C:\Users\Shay\Desktop>Blah~
Received Blah~ <<<
0

And so forth and so on.

Brian C. wrote:

(*) Incidentally: for someone to reproduce this problem, they’ll need to
know exactly what platform you have. You said “Windows Vista” but you
didn’t say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)

C:\Users\Shay\Desktop>ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

The program I used to install it (or at least what I have in my
downloads folder) is ruby186-27_rc2 from rubyinstaller.rubyforge.org.

shayhok · January 5, 2010, 10:22am

Oh, my apologies, I had put the user info there to show that I was
giving input (as if @echo hadn’t been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.

OK. I’ve now got a Windows XP box beside me. I tried the threaded
version of prg2.rb with:

(1) rubyinstaller-1.8.6-p383-rc1.exe from
rubyforge.org/projects/rubyinstaller
(this is about a year newer than the 186-27_rc2 you have)

(2) cygwin-1.7.1 which has ruby-1.8.7-p72-2 package under ‘interpreters’

(3) ruby 1.8.7-p72 binary from Download Ruby

The first fails in the way you described: that is, the sleeping thread
doesn’t “wake up” until some input has been given to stdin.

However the second and third work just fine, in the same way as it does
under Linux. That is, I see a new number every 2 seconds, and I can type
in input whenever I like and get it echoed back immediately.

So clearly the problem is with how ruby is built in the one-click
installer. You may wish to raise a ticket against it, using this as a
demonstration of the problem. But in any case there are two other
working versions of ruby you can use instead.

HTH,

Brian.

shayhok · January 5, 2010, 1:07pm

Robert K. wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr.

That can’t normally be, because the thread runs independently of the
loop.

He’s right, and I have replicated the problem. It occurs with with 1.8.6
one-click installer under Windows. With this interpreter, threads are
borked: when one thread is waiting on I/O, it prevents all other threads
from running.

The “official” 1.8.7 Windows build is OK, and so is a cygwin 1.8.7.

You can replicate the problem with this:

Thread.new do
while line = $stdin.gets
$stderr.puts “>>> Received #{line.chomp} <<<”
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end

Under the one-click installer 1.8.6, the printing loop freezes until you
give it some input.

shayhok · January 5, 2010, 1:45pm

On 01/04/2010 10:00 PM, Shay H. wrote:

called on stdin until something is provided.
Actually part of that functionality is in Ruby itself because IO uses
buffering by default. For tests like these it helps to place a line
like this at the beginning of the script:

$stdout.sync = $stderr.sync = true

As far as I can see this wasn’t done in Brian’s test scripts so
repeating with that line introduced will likely yield more accurate
results.

Kind regards

robert

shayhok · January 5, 2010, 12:45pm

On 04.01.2010 14:38, Shay H. wrote:

code of) - this program puts output into stderr, and accepts input
     stdin.puts("Received 0 five seconds ago.")
  end
end
end

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr.

That can’t normally be, because the thread runs independently of the
loop. I am rather suspecting that you are having an issue with not
waiting for all threads to proper finish or do not make sure pipes are
read properly. You could also have created a dead lock that way, i.e.
by not reading all pipes properly because a process that tries to write
to a filled pipe is blocked.

I have created a pair of test scripts which I will place below.
Strangely enough, there seems to be an issue with detecting closing of
the stderr pipe which I need to research separately. If you just invoke
the client script with “./clnt.rb 15 x” you will see the interaction
properly working. If you, for example, increase the number and omit the
reading of sin in the client the server will eventually block in one of
the puts or printf statements. It may be that this is what you are
observing.

Kind regards

robert

Place both files in the same directory.

file clnt.rb:
#!/usr/local/bin/ruby19

“client” which starts the server and every time a

zero is read from the server’s stderr pipe we send

a notification to the server’s stdin.

server’s stdout is used for logging state and is mirrored

to this process’s stdout with prefix “From server”.

require ‘open3’

for debugging

Thread.abort_on_exception = true

Open3.popen3 File.join(File.dirname($0), “serv.rb”), *ARGV do |sin,
sout, serr|
sin.sync = true

we must make sure all pipes are either read or closed!

r = Thread.new do
sout.each do |line|
puts “From server: #{line}”
end
end

threads = []

serr.each do |line|
printf “%-20s read %p\n”, Time.now, line

 line.chomp!

 if line == "0"
   threads << Thread.new do
     sleep 5
     sin.puts "Received 0 five seconds ago."
   end
 elsif /finish/i =~ line
   # interestingly enough EOF detection does not
   # work with serv.rb
   break
 end

end

puts “finished reading”

threads.each {|th| th.join}

puts “finished notifying”

init shutdown sequence

sin.close
r.join
end

file serv.rb:
#!/usr/local/bin/ruby19

“server” process which writes a fixed amount of numbers

to stderr and then closes stderr to indicate it’s finished.

first arg is no of repetitions

second arg if present says indicate end with a particular

message to stderr.

$stdout.sync = $stderr.sync = true

puts “started”

th = Thread.new do
$stdin.each do |line|
printf “%-30s read %p\n”, Time.now, line
end
end

rep = (ARGV.shift || 10).to_i
ind = ARGV.shift

rep.times do
$stderr.puts rand(3)
sleep 1
end

puts “finished writing”

indicate we are done

$stderr.puts “Finish” if ind
$stderr.close
puts “stderr.closed? #{$stderr.closed?}”

puts “finishing”

th.join

puts “finished”

shayhok · January 5, 2010, 1:50pm

On 05.01.2010 13:07, Brian C. wrote:

borked: when one thread is waiting on I/O, it prevents all other threads
end
give it some input.
For the heck of it, I’ve tested 1.8.6 and 1.9.1 compiled with MinGW:
PS C:\Scripts> ruby -v
ruby 1.9.1p243 (2009-07-16 revision 24175) [i386-mingw32]
PS C:\Scripts> c:\ruby\bin\ruby -v
ruby 1.8.6 (2009-08-04 patchlevel 383) [i386-mingw32]

Ruby 1.9.1 performs as expected: The threads start, and i gets
incremented every 2 seconds:
PS C:\Scripts> ruby .\thread.rb
-2
-1
0
1
2
3
14
1

Received 11 <<<
5
6
:q

Received :q <<<
7
8

Same script with 1.8.6:
PS C:\Scripts> c:\ruby\bin\ruby .\thread.rb
-2

Received <<<
1

Received 1 <<<
-1
2

Received 2 <<<
0
:q

Received :q <<<
1
2

(Yeah, vim user here…)

It looks like 1.8.7 introduced (or rather got backported from 1.9) a
change that makes threads work independent from the underlying OS.

shayhok · January 5, 2010, 2:39pm

Brian C. wrote:

(3) ruby 1.8.7-p72 binary from Download Ruby

Ah, that’d be it. Everything works fine now - Thank you all for your
large amount of help.

shayhok · January 5, 2010, 2:01pm

Robert K. wrote:

For tests like these it helps to place a line
like this at the beginning of the script:

$stdout.sync = $stderr.sync = true

I agree that’s a good idea, but I don’t think it actually makes a
difference here.

For one thing, the first output line ("-2") was shown on the terminal
immediately, so there’s no reason why the second line ("-1") should not
be.

For another thing, the same program worked fine on the two 1.8.7
platforms but not the 1.8.6, on the same machine.

And finally, $stderr is usually unbuffered anyway (on Unix; I know I
can’t make such assumptions about Windows).

"Dummy" IO object to push and pull data?

Required support, go to process dir

Initiate the process and its streams

Quick function for short timestamps

Read from stderr, thread stdin input

Read from stderr, thread stdin input

Thread.new do while line = $stdin.gets $stderr.puts “>>> Received #{line.chomp} <<<” end end i = -2 loop do $stderr.puts i i += 1 sleep 2 end

“client” which starts the server and every time a

zero is read from the server’s stderr pipe we send

a notification to the server’s stdin.

server’s stdout is used for logging state and is mirrored

to this process’s stdout with prefix “From server”.

for debugging

we must make sure all pipes are either read or closed!

init shutdown sequence

“server” process which writes a fixed amount of numbers

to stderr and then closes stderr to indicate it’s finished.

first arg is no of repetitions

second arg if present says indicate end with a particular

message to stderr.

indicate we are done

Thread.new do
while line = $stdin.gets
$stderr.puts “>>> Received #{line.chomp} <<<”
end
end
i = -2
loop do
$stderr.puts i
i += 1
sleep 2
end