Hi, I’ve been missing ParseTree very badly in 1.9.1. So in my naivité I
decided to try to convert YARV iseq’s to Sexp’s similar of those
produced by ParseTree. So far I can convert simple expressions, method
calls, nested method calls, arrays, hashes, I can extract method
parameter and default values provided the expression is not too
complicated.
At first it seemed easy until I hitted control structures.
Is someone aware of a similar effort?
Does anyone else craves AST manipulation in 1.9.1 as bad as me?
Anyone care to join?
Any sugested reads appart from YARV wiki?
My attempt is hosted at http://github.com/maca/iseq_parser/tree/master
Thanks
Macario
2009/8/4 Macario O. [email protected]:
[…]
Hi, I’ve been missing ParseTree very badly in 1.9.1. So in my naivité I
decided to try to convert YARV iseq’s to Sexp’s similar of those
produced by ParseTree.
[…]
Is someone aware of a similar effort?
[…]
Hello,
the Ruby 1.9.1p243 that I installed from source came with a library
called ‘ripper’ that can produce S-expression parse trees from Ruby
code, like this:
$ cat foo.rb
require ‘ripper’
require ‘pp’
pp Ripper.sexp %{
def foo
:bar
end
}
$ ruby foo.rb
[:program,
[[:def,
[:@ident, “foo”, [2, 6]],
[:params, nil, nil, nil, nil, nil],
[:body_stmt,
[[:symbol_literal, [:symbol, [:@ident, “bar”, [3, 5]]]]],
nil,
nil,
nil]]]]
I played around with that a little, but didn’t try to code anything
serious. Also, since I’m new to Ruby and haven’t used ParseTree
before, I don’t know how compatible the abstract syntax tree
representations are.
cu,
Thomas
Thomas, Thanks for you reply. I wish there was more info on ripper,
definitelly useful 
Allthough nice thing about ParseTree is that it
works with “live” objects, meanining there is no need to resort to the
source you can extract the AST for a Class even it was defined with eval
or redefined later.
Thomas C. wrote:
2009/8/4 Macario O. [email protected]:
[…]
Hi, I’ve been missing ParseTree very badly in 1.9.1. So in my naivit� I
decided to try to convert YARV iseq’s to Sexp’s similar of those
produced by ParseTree.
[…]
Is someone aware of a similar effort?
[…]
Hello,
the Ruby 1.9.1p243 that I installed from source came with a library
called ‘ripper’ that can produce S-expression parse trees from Ruby
code, like this:
$ cat foo.rb
require ‘ripper’
require ‘pp’
pp Ripper.sexp %{
def foo
:bar
end
}
$ ruby foo.rb
[:program,
[[:def,
[:@ident, “foo”, [2, 6]],
[:params, nil, nil, nil, nil, nil],
[:body_stmt,
[[:symbol_literal, [:symbol, [:@ident, “bar”, [3, 5]]]]],
nil,
nil,
nil]]]]
I played around with that a little, but didn’t try to code anything
serious. Also, since I’m new to Ruby and haven’t used ParseTree
before, I don’t know how compatible the abstract syntax tree
representations are.
cu,
Thomas
On Aug 3, 2009, at 23:20 , Macario O. wrote:
Does anyone else craves AST manipulation in 1.9.1 as bad as me?
Anyone care to join?
This is pretty awesome. It is gonna only get worse tho. 
We talked about doing this a while back and came to the conclusion
that it was more work than we were interested in investing. Please
prove us wrong and you’ll get a least a couple beers at the next
rubyconf.
A suggestion: Use test/pt_testcase.rb from the ParseTree project and
you’ll be a LOT happier. See the test cases from either ruby2c or
ruby_parser as examples of test reuse.
On 8/4/09, Thomas C. [email protected] wrote:
:bar
end
}
Hey, I’ve been wondering how one gets a tree out of ripper, or even if
it can be done at all. Thanks!
I played around with that a little, but didn’t try to code anything
serious. Also, since I’m new to Ruby and haven’t used ParseTree
before, I don’t know how compatible the abstract syntax tree
representations are.
That’s definitely not compatible with any of the ParseTree formats.
Tony A. wrote:
Does ruby_parser work on 1.9?
Yeah it does but you need to pass it ruby source code not live objects.
On Tue, Aug 4, 2009 at 12:20 AM, Macario O. [email protected]
wrote:
Does anyone else craves AST manipulation in 1.9.1 as bad as me?
Yes
Any sugested reads appart from YARV wiki?
Does ruby_parser work on 1.9?
http://parsetree.rubyforge.org/
Ryan D. wrote:
On Aug 3, 2009, at 23:20 , Macario O. wrote:
A suggestion: Use test/pt_testcase.rb from the ParseTree project and
you’ll be a LOT happier. See the test cases from either ruby2c or
ruby_parser as examples of test reuse.
Thanks for the suggestion Ryan! I don’t have much time to hack nowadays
but I love ParseTree and there is definitelly a gap in 1.9.1. As I told
I started out of naivite.
Hi Macario,
Does anyone else craves AST manipulation in 1.9.1 as bad as me?
Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
in 1.9 too, because it means that some of its Rspec matchers no longer
work. Without ParseTree (or equivalent) it’s not possible to write:
MyClass.should_not reek
Anyone care to join?
I don’t have much time available, but I’ll be happy to help.
As Ryan suggests, the starting point will be a decent suite of
regression tests.
Cheers,
Kevin
On Tue, Aug 4, 2009 at 7:56 PM, Macario O.[email protected] wrote:
Tony A. wrote:
Does ruby_parser work on 1.9?
Yeah it does but you need to pass it ruby source code not live objects.
That’s how Ripper works too. Notice in this example:
require ‘ripper’
require ‘pp’
pp Ripper.sexp %{
def foo
:bar
end
}
The %{ … } is a string.
Charles Nutter wrote:
Does ruby_parser work on 1.9?
Yeah it does but you need to pass it ruby source code not live objects.
That’s how Ripper works too. Notice in this example:
require ‘ripper’
require ‘pp’
pp Ripper.sexp %{
def foo
:bar
end
}
The %{ … } is a string.
I think that’s the point You have to pass it a String source for
it to parse. You can’t take an object which Ruby has already parsed and
installed, such as method(:foo) in this case, and get the sexp from the
object.
Kevin Rutherford wrote:
Hi Macario,
Does anyone else craves AST manipulation in 1.9.1 as bad as me?
Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
in 1.9 too, because it means that some of its Rspec matchers no longer
work. Without ParseTree (or equivalent) it’s not possible to write:
MyClass.should_not reek
You can still pull the code for a method using #source_location [though
it’s definitely not as nice as using parse tree was, and not as stable].
A crude example of extracting a method can be found in the method
describer gem [1].
What problems are you running up against exactly? Learning lessons from
python decompyle might be helpful, too.
I also remember that Paul B. had some type of iseq → c translator,
that might be related.
GL!
-r
[1] http://github.com/rogerdpack/method_describer/tree/master
On Wed, Aug 5, 2009 at 9:56 AM, Brian C.[email protected]
wrote:
I think that’s the point You have to pass it a String source for
it to parse. You can’t take an object which Ruby has already parsed and
installed, such as method(:foo) in this case, and get the sexp from the
object.
Yeah, what’s wrong with that?
Ignoring for the moment that different impls (and even different revs
of MRI) have (often wildly) different AST structures…
The problem with expecting you can get an AST at runtime is that it
means impls have to hold on to the AST for you. Guess what one of the
largest startup memory costs is in MRI and JRuby…that’s right, it’s
the AST from loading all that code into a big in-memory data
structure. Ruby 1.9 doesn’t have the AST available anymore for a good
reason: it doesn’t need it. JRuby won’t need it soon either once our
new compiler lands, and already doesn’t have it available when you
precompile code. So I guess it’s a question of which is
better…getting rid of that memory use or having the AST around all
the time.
And again, that ignores that the AST is ever-changing and usually very
different across impls.
Hi Roger. nice to read you.
I’ve allready done that with our named arguments gem but I wan’t to have
AST for live objects. The project is going dormant for a while
(allthough is not gonna die) because lack of time but I found great
advice in this thread:
Having a tight and organized set of tests and recycling tests from other
proyects such as PT and checking other proyects that work with bytecode.
Cheers
Macario
Roger P. wrote:
Kevin Rutherford wrote:
Hi Macario,
Does anyone else craves AST manipulation in 1.9.1 as bad as me?
Reek (http://github.com/kevinrutherford/reek/tree) will miss ParseTree
in 1.9 too, because it means that some of its Rspec matchers no longer
work. Without ParseTree (or equivalent) it’s not possible to write:
MyClass.should_not reek
You can still pull the code for a method using #source_location [though
it’s definitely not as nice as using parse tree was, and not as stable].
A crude example of extracting a method can be found in the method
describer gem [1].
What problems are you running up against exactly? Learning lessons from
python decompyle might be helpful, too.
I also remember that Paul B. had some type of iseq → c translator,
that might be related.
GL!
-r
[1] http://github.com/rogerdpack/method_describer/tree/master
On Sun, Aug 9, 2009 at 3:24 AM, Charles Oliver
Nutter[email protected] wrote:
On Wed, Aug 5, 2009 at 9:56 AM, Brian C.[email protected] wrote:
And again, that ignores that the AST is ever-changing and usually very
different across impls.
I guess if I could really chose I would like a switch, and would like
the VM to keep the AST iff that switch is set (-a|–ast).
Goo{df} idea?
Cheers
Robert
Brian C. wrote:
…
I think that’s the point You have to pass it a String source for
it to parse. You can’t take an object which Ruby has already parsed and
installed, such as method(:foo) in this case, and get the sexp from the
object.
One way to generate AST from live object is to use method_missing, which
is applicable in 1.9 with pure ruby code.
see
http://rubyquiz.com/quiz95.html
Charles Nutter wrote:
On Wed, Aug 5, 2009 at 9:56 AM, Brian C.[email protected]
wrote:
I think that’s the point You have to pass it a String source for
it to parse. You can’t take an object which Ruby has already parsed and
installed, such as method(:foo) in this case, and get the sexp from the
object.
Yeah, what’s wrong with that?
Ignoring for the moment that different impls (and even different revs
of MRI) have (often wildly) different AST structures…
The problem with expecting you can get an AST at runtime is that it
means impls have to hold on to the AST for you. Guess what one of the
largest startup memory costs is in MRI and JRuby…that’s right, it’s
the AST from loading all that code into a big in-memory data
structure. Ruby 1.9 doesn’t have the AST available anymore for a good
reason: it doesn’t need it. JRuby won’t need it soon either once our
new compiler lands, and already doesn’t have it available when you
precompile code. So I guess it’s a question of which is
better…getting rid of that memory use or having the AST around all
the time.
And again, that ignores that the AST is ever-changing and usually very
different across impls.
I see the AST representation spitted by ParseTree as some sort of lisp
like meta programming language in its own right (I’ve never used lisp
but I am increasingly interested in it). I really don’t care if it
corresponds to Ruby internal representation as long as I can use it to
generate Ruby code (or not Ruby code) and it provides a level of
introspection not available without access to it.
I think I once saw a discussion between Ola B., Mentalguy and others
about the need to have a common sexp representation across diferent
implementations, some sort or Risp.
I think there is a gap in 1.9.x that RubyParser fills partialy as it
only works with static code.
Thanks
Macario
Lui Core wrote:
I think that’s the point You have to pass it a String source for
it to parse. You can’t take an object which Ruby has already parsed and
installed, such as method(:foo) in this case, and get the sexp from the
object.
One way to generate AST from live object is to use method_missing, which
is applicable in 1.9 with pure ruby code.
see
Ruby Quiz - Code to S-Exp (#95)
That’s not proper Ruby; I don’t see any loops or conditional branches. I
suspect this approach requires you to solve the halting problem. e.g.
sxp {
for i in 3…1.0/0
for j in 3…i
for k in j…i
for p in 3…i
if ip + jp == k**p
puts “fermat disproved: #{i}, #{j}, #{k}, #{p}”
break
end
end
end
end
end
}
Robert D. wrote:
On Sun, Aug 9, 2009 at 3:24 AM, Charles Oliver
Nutter[email protected] wrote:
On Wed, Aug 5, 2009 at 9:56 AM, Brian C.[email protected] wrote:
And again, that ignores that the AST is ever-changing and usually very
different across impls.
I guess if I could really chose I would like a switch, and would like
the VM to keep the AST iff that switch is set (-a|–ast).
Goo{df} idea?
Unfortunately, by then it will already be too late. There’s nothing
the VM can do about it. That’s exactly the reason why ParseTree
doesn’t work on YARV: the only component that needs the parsetree is
the compiler. By the time the compiler hands off the code to the VM,
the parsetree is long gone. The same applies to XRuby and Ruby.NET as
well as the AOT modes of Rubinius, JRuby, IronRuby and MacRuby.
And remember that the compiler and the VM need not be run by the same
person, on the same machine or at the same time. In HotRuby, for
example, the compiler is generally run by the web developer on the web
server at development time whereas the VM is run by the user inside
the browser on the client, maybe months after the code was compiled.
jwm
One way to generate AST from live object is to use method_missing, which
is applicable in 1.9 with pure ruby code.
see
Ruby Quiz - Code to S-Exp (#95)
That is fascinating. Maybe I should read the ruby quizzes
-r