Reading stdout & stderr from a pipe with popen3

Ruby
1

I originally posted this to a Ruby forum at railsforum.com, but I just
found this list so thought I’d try here since I haven’t gotten a
response at railsforum.

OK…so I’m using popen3 to run some command line stuff and be able to
get back stdout/stderr…

$cmdin, $cmdout, $cmderr = Open3.popen3(“zmprov”)

zmprov opens a new command prompt to which I can send more commands to
and get back output…

$cmdin.puts(“sm username”)
$cmdin.puts(“sm”)
count = 0
$cmdout.each |line|
count += 1 if line.include?(“\n”)
puts line
break if count == 2 # IF I FIND 2 BLANK LINES IN stdout, BREAK THE
LOOP
end

Which works just fine. However, I’ve run into a snag. $cmdout and
$cmderr are pipes that never close until I send the “quit” command…so
in order to break out of my loops I’ve got to jump through some extra
hoops to look for specific patterns in the output. In the above example
I have to send an extra “sm” command without the username, and in that
output I can find the two blank lines…so I know it’s done and can drop
out of the loop.

It’s a pain, but I can do that. The real problem becomes what to do
when a command returns something in stderr? Because stdout returns
nothing…there’s nothing to check for to drop out of that loop before I
check for stderr…check the example below.

$cmdin.puts(“sm BADUSERNAME”) # PRODUCES AN ERROR
$cmdout.each |line|
puts line
end
$cmderr.each |line|
puts line
end

If the command produces an error…I never get out of the stdout loop
to grab the error because stdout is returning me nothing…it’s just an
open pipe with no data in it until I run a command that produces stdout.
So I never get where I can read stderr.

A coworker suggested I run the original “zmprov” command as “zmprov
2>&1” so that stderr is returned in stdout. Which will work…but I was
just wondering if I could get the code to work as designed. If stdout
is returning data…read it and do something…if stderr is returning
data then do something with that. I’m wanting to keep one from blocking
my ability to read the other.

After some further research and a little testing, I’ve found that using
IO.readpartial
(http://www.noobkit.com/show/ruby/ruby/ruby-core/io/readpartial.html)
might be what I’m looking for, but I’m not sure. So I can do something
like…

$cmdin.readpartial(4096)

…and it will grab all the data in the pipe (up to 4096 bytes I think)
and when it runs out of stuff to read it returns control back to the
program. What happens though if there is more data waiting in the pipe
than I call for with my IO.readpartial(maxlen) argument? Will it still
return control or will it keep reading 4096 blocks until it runs out and
THEN return control? I think that is how I would want it to handle.

There is almost no examples out there about IO.readpartial, so I’m
hoping someone here can help point me in the right direction.

Thanks,
Matt

2

bump…

Just trying to see if anyone knows IO.readpartial better than me.

Matt

----- Original Message -----
From: “Matt M.” [email protected]
To: “ruby-talk ML” [email protected]
Sent: Thursday, July 3, 2008 3:07:15 PM GMT -06:00 US/Canada Central
Subject: Reading stdout & stderr from a pipe with popen3

I originally posted this to a Ruby forum at railsforum.com, but I just
found this list so thought I’d try here since I haven’t gotten a
response at railsforum.

OK…so I’m using popen3 to run some command line stuff and be able to
get back stdout/stderr…

$cmdin, $cmdout, $cmderr = Open3.popen3(“zmprov”)

zmprov opens a new command prompt to which I can send more commands to
and get back output…

$cmdin.puts(“sm username”)
$cmdin.puts(“sm”)
count = 0
$cmdout.each |line|
count += 1 if line.include?(“\n”)
puts line
break if count == 2 # IF I FIND 2 BLANK LINES IN stdout, BREAK THE
LOOP
end

Which works just fine. However, I’ve run into a snag. $cmdout and
$cmderr are pipes that never close until I send the “quit” command…so
in order to break out of my loops I’ve got to jump through some extra
hoops to look for specific patterns in the output. In the above example
I have to send an extra “sm” command without the username, and in that
output I can find the two blank lines…so I know it’s done and can drop
out of the loop.

It’s a pain, but I can do that. The real problem becomes what to do
when a command returns something in stderr? Because stdout returns
nothing…there’s nothing to check for to drop out of that loop before I
check for stderr…check the example below.

$cmdin.puts(“sm BADUSERNAME”) # PRODUCES AN ERROR
$cmdout.each |line|
puts line
end
$cmderr.each |line|
puts line
end

If the command produces an error…I never get out of the stdout loop
to grab the error because stdout is returning me nothing…it’s just an
open pipe with no data in it until I run a command that produces stdout.
So I never get where I can read stderr.

A coworker suggested I run the original “zmprov” command as “zmprov
2>&1” so that stderr is returned in stdout. Which will work…but I was
just wondering if I could get the code to work as designed. If stdout
is returning data…read it and do something…if stderr is returning
data then do something with that. I’m wanting to keep one from blocking
my ability to read the other.

After some further research and a little testing, I’ve found that using
IO.readpartial
(http://www.noobkit.com/show/ruby/ruby/ruby-core/io/readpartial.html)
might be what I’m looking for, but I’m not sure. So I can do something
like…

$cmdin.readpartial(4096)

…and it will grab all the data in the pipe (up to 4096 bytes I think)
and when it runs out of stuff to read it returns control back to the
program. What happens though if there is more data waiting in the pipe
than I call for with my IO.readpartial(maxlen) argument? Will it still
return control or will it keep reading 4096 blocks until it runs out and
THEN return control? I think that is how I would want it to handle.

There is almost no examples out there about IO.readpartial, so I’m
hoping someone here can help point me in the right direction.

Thanks,
Matt

3

On Jul 3, 2008, at 2:07 PM, Matt M. wrote:

A coworker suggested I run the original “zmprov” command as “zmprov
2>&1” so that stderr is returned in stdout. Which will work…but I
was just wondering if I could get the code to work as designed. If
stdout is returning data…read it and do something…if stderr is
returning data then do something with that. I’m wanting to keep one
from blocking my ability to read the other.

the issue is even worse than you describe, the program can easily
become blocked if it’s stdout or stderr pipes get full - to fix the
situation you need to use threads, one processing both stdout and
stderr asynchronously where each may have to trigger actions on
stdin. the general pattern is

q = Queue.new

err = Thread.new do
Thread.current.abort_on_exception = true

 while(( line = stderr.gets ))
   ...
   q.push :somthing if some_condition_on(line)
 end
 q.push :stderr_done

end

out = Thread.new do
Thread.current.abort_on_exception = true

 while(( line = stdout.gets ))
   ...
   q.push :something if some_condition_on(line)
 end
 q.push :stdout_done

end

in = Thread.new do
Thread.current.abort_on_exception

 while(( command = q.pop ))
   ...
   break if stdout_done and stderr_done
 end

end

in.join

so basically have one thread sending commands down stdin. start a
thread each for stdout and stderr, each doing their own processing, if
they encounter something which means input needs to be send push it
onto a queue to allow the stdin thread to do it on their behave. this
of course ignores exceptional conditions and coordination between the
stdout and stderr threads, but it’s one approach.

the big conditions any solution needs to handle are having no output
on either stderr or stdout or being blocked on a write to either due a
full pipe, which is why this cannot work safely:

loop do
handle stdout.gets
handle stderr.gets
end

check out open4 and session for examples of using threads to process
both stdout and stderr concurrently.

http://codeforpeople.com/lib/ruby/

gem install open4 session

cheers.

a @ http://codeforpeople.com/

4

OK…using what ara.t.howard suggested, I’ve rewritten a block of my
code and it looks something like this…

CODE
require “open3”
require “thread”

def runner(stdin, stdout, stderr, cmd)
queue = Queue.new
stdin.puts(cmd)
errthd = Thread.new do
Thread.current.abort_on_exception = true
while(( line = stderr.gets ))
queue.push(line)
end
queue.push :stderr_done
end

outthd = Thread.new do
Thread.current.abort_on_exception = true
while(( line = stdout.gets ))
queue.push(line)
end
queue.push :stdout_done
end

inthd = Thread.new do
Thread.current.abort_on_exception = true
while(( stuff = queue.pop ))
puts stuff
break if :stdout_done and :stderr_done
end
end

inthd.join
errthd.exit
outthd.exit
end

stdin, stdout, stderr = Open3.popen3(“zmprov”)
runner(stdin, stdout, stderr, “selectMailbox shares”)
runner(stdin, stdout, stderr, “getAllFolders”)
<<<<CODE

It almost works. I found that if I didn’t add the errthd.exit and
outthd.exit lines, the program would get stuck on inthd.join(waiting for
the break conditions?) most of the time…very rarely would it complete.
But the output is always incomplete. If I run the commands directly
from the command line it look something like this…

GOOD OUTPUT
$ zmprov
prov> selectMailbox shares
mailbox: [email protected], size: 174.36 KB, messages: 115, unread: 59
mbox [email protected]> getAllFolders
Id View Unread Msg Count Path

     1  conv           0           0  /
    16  docu           0           0  /Briefcase
    10  appo           0           0  /Calendar
    14  mess           0           0  /Chats
     7  cont           0           0  /Contacts
     6  mess           0           0  /Drafts
    13  cont           0           2  /Emailed Contacts
   257  appo           0           0  /evite
     2  mess           0           0  /Inbox
     4  mess           0           0  /Junk
    12  wiki           0           0  /Notebook
     5  mess           0           2  /Sent
    15  task           0           0  /Tasks
     3  conv           0           0  /Trash

<<<<GOOD OUTPUT

When I run my program though it looks more like this…

BAD OUTPUT
prov> mailbox: [email protected], size: 174.36 KB, messages: 115,
unread: 59
mbox [email protected]> Id View Unread Msg Count Path
<<<<<BAD OUTPUT

The first line always seems to come out right… but the ‘getAllFolders’
command never comes out correctly. Maybe 1 in 10 times it will return a
line or two more than just the ‘mbox’ line, but most of the time it’s
just that line of the output…the first line returned from the
‘getAllFolders’ command. So the thread for standard out doesn’t seem to
be grabbing all the data out of the pipe? I tried adding a short pause
‘sleep 2’ above the threads to see if maybe there was some slow response
from them getting in the way, but that didn’t really change anything.

Any help or ideas anyone can offer?

Thanks,
Matt

5

On Jul 17, 2008, at 10:53 AM, Matt M. wrote:

def runner(stdin, stdout, stderr, cmd)
queue = Queue.new
stdin.puts(cmd)

stdin.flush

Thread.current.abort_on_exception = true
break if :stdout_done and :stderr_done
this test reads “break if true and true” !?!

you are testing truth on two symbols. you need to keep vars and set
them when :stdout_done and :stderr_done are seen on the queue.

if stuff == :stdout_done
stdout_done == true
next
end

if stuff == :stderr_done
stderr_done = true
next
end

if stdout_done and stderr_done
end

etc…

btw - you are simply re-writing the open4 gem - why not use it?

end
end

inthd.join
errthd.exit
outthd.exit
end

a @ http://codeforpeople.com/

6

btw - you are simply re-writing the open4 gem - why not use it?

Because I already kind of knew Open3 and it took me long enough to
figure out how to use that one…and I like to make my life as
difficult as possible. :slight_smile:

So now I’m trying Open4 as you suggest and have a couple questions.

require “rubygems”
require “open4”

pid, stdin, stdout, stderr = Open4::popen4 “zmprov”
stdin.puts “sm shares”
stdin.puts “gaf”
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

The check for stderr being empty? doesn’t work…it prints a blank line
if no error was returned so I tried checking for newline(’\n’) but it
doesn’t work either. What do I check for if nothing was returned on
stderr?

The other thing I noticed is that I have to include the stdin.close or
the program stops. I guess it’s waiting for the close? It can’t be
after I try to print stderr/stdout either…it has to be before I check
for those. So I guess I have to send all my input and then close the
input and then check the statuses of stdout/stderr to decide how to
proceed?

Looks like this will work and there doesn’t seem to be any blocking
getting in the way. Eventually I’m going to be sending thousands of
these commands to the zmprov> prompt so I want this to be a tight and
fast as possible. As long as I can get the full stdout/stdin of the
last command before moving on to the next account…I think this will
work.

Thanks,
Matt

7

Well drat…

pid, stdin, stdout, stderr = Open4::popen4 “zmprov”
stdin.puts “sm shares”
stdin.puts “gaf”
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
stdin.puts “sm shares2” # line 52
stdin.puts “gaf”
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

Trying to do that doesn’t work, it says the stream is closed and I can’t
send anything new to it.
./pipe_test.rb:52:in `write’: closed stream (IOError)

So I don’t want to stdin.close until I’m done processing a bunch of
accounts, but I need to check stdout/stderr after I check each account
without the stdin.close. Is that possible?

Matt

----- Original Message -----
From: “Matt M.” [email protected]
To: “ruby-talk ML” [email protected]
Sent: Thursday, July 17, 2008 1:13:40 PM GMT -06:00 US/Canada Central
Subject: Re: Reading stdout & stderr from a pipe with popen3

btw - you are simply re-writing the open4 gem - why not use it?

Because I already kind of knew Open3 and it took me long enough to
figure out how to use that one…and I like to make my life as
difficult as possible. :slight_smile:

So now I’m trying Open4 as you suggest and have a couple questions.

require “rubygems”
require “open4”

pid, stdin, stdout, stderr = Open4::popen4 “zmprov”
stdin.puts “sm shares”
stdin.puts “gaf”
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

The check for stderr being empty? doesn’t work…it prints a blank line
if no error was returned so I tried checking for newline(‘\n’) but it
doesn’t work either. What do I check for if nothing was returned on
stderr?

The other thing I noticed is that I have to include the stdin.close or
the program stops. I guess it’s waiting for the close? It can’t be
after I try to print stderr/stdout either…it has to be before I check
for those. So I guess I have to send all my input and then close the
input and then check the statuses of stdout/stderr to decide how to
proceed?

Looks like this will work and there doesn’t seem to be any blocking
getting in the way. Eventually I’m going to be sending thousands of
these commands to the zmprov> prompt so I want this to be a tight and
fast as possible. As long as I can get the full stdout/stdin of the
last command before moving on to the next account…I think this will
work.

Thanks,
Matt

8

On Jul 17, 2008, at 12:22 PM, Matt M. wrote:

stdin.close
accounts, but I need to check stdout/stderr after I check each
account without the stdin.close. Is that possible?

Matt

it’s entirely up to the client program - this has nothing to do with
open4 or any other program.

you’ve got all sorts if issues happening here

  • you simply need to flush stdin, not close it, if you plan to
    continue to use it
  • you are reading to the end of stdout and stderr, this blocks until a
    program exits, which will not happen until you close stdin, which is
    why you are needing to do that to see any output
  • zmprov may or may not buffer stdout and stderr, i have no idea how
    this program is written an so cannot say

basically, if we assume zmprov is written in a line oriented fashion
you might be able to do something like

stdin.puts input

output = stdout.gets output

if output =~ /error/
…

but i see you are checking stderr too, so you have a problem: you have
to read from both stdout and stderr and one, or both, might be empty.
you program is going to block until you check both. so you have options

  • use threads and a queue to check both, develop your own protocol
    using the code from open4 as an example of how to read from two io
    objects simultaneously
  • use select to read from either stdout or stderr, whichever becomes
    available
  • re-write zmprov so send all output on stderr and check only that

in short your question is impossible to answer without understanding
what, exactly zmprov does and how it produces output on stdout and
stderr. it’s also impossible to solve without a solid understanding
of how io and, specifically io buffering works.

hope that helps.

a @ http://codeforpeople.com/