Gritty Details of super()

Ruby
1

Hey guys,

I thought I understood super just fine until I ran into a little issue
recently. It’s not really documented in ruby-doc as it’s a keyword and
such… and the explanations I’ve received vary widely.

Lots of people say: "Oh, super just calls the method of the same name in
the parent class (an instance method for an instance of a class, or a
singleton method if an instance of Class or Module), and it just goes up
the inheritance chain until up to BasicObject or until it finds the
method name and then if nothing is found it raises an error.

Also, self is set to the scope of self in the method originally calling
super() and an error is raised there as well if nothing is found.

Fine…

But the issue I ran into was calling super in a singleton method of a
class object ie: Someclass.somemethod(), and hoping that my included
module that also defined a singleton method would catch the call to
super().

It turns out that module objects don’t catch the super call, and they
also don’t catch method_missing and other hooks if they’re included but
still define singleton methods.

Why is this?

I’ve been thinking about it, and is it because the anonymous class that
gets put into the inheritance chain doesn’t get the singleton method at
all… like since only the instance methods, class variables and
constants are in that anon. class?

And further, is it because the anonymous class is skipped entirely in
the call to a class’s .superclass() method? And super only goes directly
to the superclasses to check the method?

I have lots of questions, and an answer as to what’s possible and what’s
not, but I’d really like an explanation as to why this is happening this
way.

Simple illustrative code snippet:

Thanks for taking the time,

-Luke

2

On Aug 21, 2011, at 11:21 AM, luke gruber [email protected] wrote:

But the issue I ran into was calling super in a singleton method of a
class object ie: Someclass.somemethod(), and hoping that my included
module that also defined a singleton method would catch the call to
super().

This isn’t related to super so much as it is the nature of inheritance
and modules in Ruby. See this diagram:

If you want inheritance for singleton methods of an object from a module
you must extend the object with a module defining instance methods.

For example:

module Foo
def self.included(klass)
klass.extend(ClassMethods)
end
module ClassMethods
def bar; …; end
end
end

class Bar
include Foo
end

Bar.bar

(typed on the phone; forgive untested errors)

3

On Sun, Aug 21, 2011 at 7:48 PM, Gavin K. [email protected] wrote:

end
end

class Bar
include Foo
end

Bar.bar

(typed on the phone; forgive untested errors)

Direct extending works as well

 1  #!/usr/bin/env ruby
 2
 3  module Mod
 4
 5    def who
 6      "mod"
 7    end
 8
 9  end
10
11  class Test
12    extend Mod
13
14    def self.who
15      ret = super
16      [ret, "test"]
17    end
18
19  end
20
21  p Test.ancestors  # includes Mod
22  p Test.superclass # is Object!!
23  p Test.who # DOESN'T WORK

Kind regards

robert

4

extend Mod

Thanks Gavin and Robert.

Yeah, I knew that extend would do the trick as it includes the instance
methods of the module in the singleton class of the extended object, but
was wondering this: if you include a module that defines a singleton
method, and since the included module ‘effectively becomes the
superclass of whatever class included it (pickaxe)’, then calls to super
should resolve at the ‘effective superclass’, no?

On looking into it further though, when a module is included it isn’t
‘literally’ a superclass, as in all calls to Child.superclass will not
resolve at the included module, even though that module, upon inclusion,
created an anonymous class directly above the class that included the
module.

In the example that Robert gave above, on line 22

22 p Test.superclass

Right now it’s Object, which I understand. But if we also include Mod,
wouldn’t that make the superclass of Test some anonymous class that
proxies to Mod?

But it doesn’t… It’s still Object.

So I guess what’s weird is that Ruby is creating a superclass when we
include a module, but that superclass is skipped with calls to
superclass.

And when people say that super goes looking in the superclass, that is
not true. It looks in the ancestors, because clearly super can resolve
to anonymous superclasses (included modules, extended modules) as in the
example that Robert gave.

hehe, if anyone is still reading this rambling stuff, I appreciate it.

-Luke

5

luke gruber wrote in post #1017749:

In the example that Robert gave above, on line 22

22 p Test.superclass

Right now it’s Object, which I understand. But if we also include Mod,
wouldn’t that make the superclass of Test some anonymous class that
proxies to Mod?

But it doesn’t… It’s still Object.

Remember, the class ruby creates when you include a module is
‘anonymous’. That means it has no name. As a result, superclass()
doesn’t list it.

6

luke gruber wrote in post #1017749:

hehe, if anyone is still reading this rambling stuff, I appreciate it.

I understood the issue you brought up in your first post. The following
is an explanation of the issue.

Suppose you have this hierarchy of classes:

class Object
def Object.meditate
puts “Ohhhmmmmm”
end
end

class Cat
def Cat.species
puts “Siamese”
end

def greet
puts ‘meow’
end
end

class Dog < Cat
end

You can test out the hierarchy with this code:

d = Dog.new
d.greet #meow
Dog.species #Siamese
Dog.meditate #Ohhmm

Here is what the method lookup looks like:

‘#’ = singleton class, which contains the singleton methods

Object -> #Object
^ ^
| |
Cat --> #Cat
^ ^
| |
Dog --> #Dog (methods called on Dog are looked up
^ in #Dog, #Cat, #Object)
|
d => #d (methods called on d are looked up in #d, Dog, Cat, Object)

Now lets include a module in Dog:

class Object
def Object.meditate
puts “Ohhhmmmmm”
end
end

class Cat
def Cat.species
puts “Siamese”
end

def greet
puts ‘meow’
end
end

module M
def greet
puts “Hi. I’m M.”
end
end

class Dog < Cat
include M
end

Now we get different output for d.greet:

d = Dog.new
d.greet #Hi. I’m M
Dog.species #Siamese
Dog.meditate #Ohhmmm

The typical explanation for that behavior is that the module M becomes
an anonymous class that ruby inserts directly above Dog in the
inheritance chain:

‘#’ = singleton class

   Object -> #Object
      ^        ^
      |        |

greet() Cat --> #Cat
^
|
-
greet() | M |
-
^ ^
| |
Dog --> #Dog
^
|
d => #d

So when ruby looks up greet() for d, it finds the greet() method in M
before the greet() method in Cat.

Now if the anonymous class (M) behaved like the other classes in
the inheritance chain, you would also expect this to happen:

‘#’ = singleton class

Object -> #Object
^ ^
| |
Cat --> #Cat
^ ^
| |
- |
| M | -> #M
-
^ ^
| |
Dog --> #Dog
^
|
d => #d

But if you add a singleton method to M, you can verify that that doesn’t
happen:

class Object
def Object.meditate
puts “Ohhhmmmmm”
end
end

class Cat
def Cat.species
puts “Siamese”
end

def greet
puts ‘meow’
end
end

module M
def M.meditate
puts “MMMMMMM”
end

def greet
puts “Hi. I’m M.”
end
end

class Dog < Cat
include M
end

Dog.meditate #Ohhhmmmmm

The question is: Why isn’t M’s singleton class inserted into the
singleton class hierarchy?

People who are interested in such things have probably read that
including a module in a class does not include the singleton methods of
the module in the class. What that means is, #M (= the singleton
class, which contains M’s singleton methods) does not get inserted
between #Dog and #Cat.

As to why that is? I have no idea. Things would be nice and
symmetrical if a module’s singleton class did get inserted in the
singleton class hierarchy.

(The method lookup diagrams were copied from Metaprogramming Ruby.)

7

On Sun, Aug 21, 2011 at 10:26 PM, luke gruber [email protected]
wrote:

extend Mod

Thanks Gavin and Robert.

You’re welcome.

Yeah, I knew that extend would do the trick as it includes the instance
methods of the module in the singleton class of the extended object, but
was wondering this: if you include a module that defines a singleton
method, and since the included module ‘effectively becomes the
superclass of whatever class included it (pickaxe)’, then calls to super
should resolve at the ‘effective superclass’, no?

Care must be taken not to confuse inheritance hierarchies: there are
two of them as 7stud has shown with the ASCII graphic. Including a
module in a class inserts it into its chain of ancestors, but has no
effect on the inheritance hierarchy of the class’s singleton class:

Basic class:

irb(main):001:0> class C;end
=> nil
irb(main):002:0> C.ancestors
=> [C, Object, Kernel, BasicObject]
irb(main):003:0> sc = class <<C;self;end
=> #Class:C
irb(main):004:0> sc.ancestors
=> [Class, Module, Object, Kernel, BasicObject]

With module included:

irb(main):005:0> module M;end
=> nil
irb(main):006:0> class C;include M;end
=> C
irb(main):007:0> C.ancestors
=> [C, M, Object, Kernel, BasicObject]
irb(main):008:0> sc.ancestors
=> [Class, Module, Object, Kernel, BasicObject]

Note how output from lines 2 and 7 differs but not from lines 4 and 8.

wouldn’t that make the superclass of Test some anonymous class that
proxies to Mod?

But it doesn’t… It’s still Object.

Another confusion that may arise: there are two relations for classes,
a superclass relation and an inheritance relation. The superclass
relation only ever includes classes and it is fixed at class creation
time (by stating “class X < Y” or “Class.new Y”) but the inheritance
relation includes classes and modules and is modified whenever a
module is included somewhere in the class hierarchy. Only the
inheritance relation is considered for method lookups.

Note: the naming “superclass relation” and “inheritance relation” is
not official lingo, I just made it up to have distinguishable names
for the two.

So I guess what’s weird is that Ruby is creating a superclass when we
include a module, but that superclass is skipped with calls to
superclass.

A module cannot be a super/class/.

And when people say that super goes looking in the superclass, that is
not true. It looks in the ancestors, because clearly super can resolve
to anonymous superclasses (included modules, extended modules) as in the
example that Robert gave.

Exactly!

hehe, if anyone is still reading this rambling stuff, I appreciate it.

It can be confusing at times, especially since everything ultimately
inherits Object. This means especially that defining an instance
method of Object makes it available to everything, i.e. instances,
classes and singleton classes. This can sometimes confuse inheritance
understanding as well.

Kind regards

robert

8

The question is: Why isn’t M’s singleton class inserted into the
singleton class hierarchy?

As to why that is? I have no idea. Things would be nice and
symmetrical if a module’s singleton class did get inserted in the
singleton class hierarchy.

(The method lookup diagrams were copied from Metaprogramming Ruby.)

7stud

Thanks 7stud, that was exactly what I was trying to get at.
The confusion I have with the current approach is this:

If Ruby wanted us to forget about the whole anonymous superclass
proxying the module on inclusion or extension, then in order for the
module (which, in our mental makeup in this case is right above the
class that included/extended the module), it would make the most sense
for singleton methods called on our class to resolve/lookup the module’s
singleton methods upon a method_missing or super or whatever.

However, this is not the case as we all know.

So OK, maybe Ruby wants us to KNOW about the anonymous superclass that
is proxying the module, and for this to not only be an implementation
feature/hack, but a feature of the language that we can put in our
mental model of method lookup. But if we put this in our mental model,
then a couple of problems arise for me.

First, ::ancestors. ::ancestors shows the module name, which is nice,
but crumples our mental model a bit as it should show the anon
superclass if our mental model was changed. The second
problem/misunderstanding for me is that the anonymous superclass that
proxies the module (ASCTPTM…hehe)
does not proxy the singleton methods of the module. This for me,
actually makes sense now. Of course it wouldn’t, because only instance
methods, CONSTANTS and @@variables are looked up.

So with all that said, I guess I’m trying to say that, in my mind, Ruby
is a little confused as to which mental model people are supposed to
take, and in some cases these two mental models conflict a little.

I’d love to hear your thoughts.

Oh, and I’ll reply to Robert later as he had some great points.

Thanks all,

-Luke

9

The superclass relation only ever includes classes and it is fixed at >class
creation time (by stating “class X < Y” or “Class.new Y”) but the >inheritance
relation includes classes and modules and is modified whenever >a module is
included somewhere in the class hierarchy. Only the
inheritance relation is considered for method lookups.

Robert

That’s a great way of putting it Robert. Thanks for that.

Another, sort of related thing:

Speaking of method lookup, class objects that have singleton classes
embedded in singleton classes look up methods in the superclass’s
singleton class’s singleton class (if it exists) and actually stop at
the top-most singleton class of the original class object. A mouthful…
I find this really cool if not pretty useless :slight_smile: And like if a class
object has 10 singleton classes with a greet() method defined in the
seventh and eighth, and the subclass calls greet() in its sixth
singleton class, it won’t resolve.

Umm… I Just read that over and wet myself.

Illustrative code snippet:

Thanks all

-Luke

10

I am working on RHEL 5. I have to build a web automation framwork using
Ruby. when I am trying to read a XML file using Chilkat ruby I am
getting following error.

Ruby code where I am getting the error.

require ‘rubygems’
require ‘Chilkat’

…

…

xml = Chilkat::CkXml.new()
xml.LoadXmlFile(’/root/Desktop/XML/Vendor_Registration_Flow01.xml’)
@strContent = Chilkat::CkString.new()
@strTag = Chilkat::CkString.new()
@strXml = Chilkat::CkString.new()
@processNode = xml.SearchForTag(nil,“t:test_case”)

…

…

While running the code I am getting this message –

/root/Desktop/readxml.rb:14:in LoadXmlFile': Expected argument 0 of type CkXml *, but got Chilkat::CkXml #<Chilkat::CkXml:0x000000038e3... (TypeError) in SWIG method 'LoadXmlFile' from /root/Desktop/readxml.rb:14:in’

Please help me in resolving the error.

Thanks.

11

luke gruber wrote in post #1017884:

The question is: Why isn’t M’s singleton class inserted into the
singleton class hierarchy?

As to why that is? I have no idea. Things would be nice and
symmetrical if a module’s singleton class did get inserted in the
singleton class hierarchy.

(The method lookup diagrams were copied from Metaprogramming Ruby.)

7stud

Thanks 7stud, that was exactly what I was trying to get at.
The confusion I have with the current approach is this:

If Ruby wanted us to forget about the whole anonymous superclass
proxying the module on inclusion or extension, then in order for the
module (which, in our mental makeup in this case is right above the
class that included/extended the module), it would make the most sense
for singleton methods called on our class to resolve/lookup the module’s
singleton methods upon a method_missing or super or whatever.

However, this is not the case as we all know.

So OK, maybe Ruby wants us to KNOW about the anonymous superclass that
is proxying the module, and for this to not only be an implementation
feature/hack, but a feature of the language that we can put in our
mental model of method lookup. But if we put this in our mental model,
then a couple of problems arise for me.

First, ::ancestors. ::ancestors shows the module name, which is nice,
but crumples our mental model a bit as it should show the anon
superclass if our mental model was changed.

I don’t think changing the name ‘M’ to ‘anon-M’, or even worse: “”,
would make much difference. You just have to know that superclass()
doesn’t return included modules, and ancestors does.

The second
problem/misunderstanding for me is that the anonymous superclass that
proxies the module (ASCTPTM…hehe)
does not proxy the singleton methods of the module.

I think you aren’t making the correct distinction here: there are two
classes involved–not one. In all situations in ruby, the singleton
methods are not in the same
class as the instance methods. For instance, if you do this:

class Dog
def Dog.greet
puts ‘hello’
end

def speak
puts ‘woof, woof’
end
end

That doesn’t create just one class. ruby creates two classes: Dog and
#Dog, which are in two different lookup hierarchies–as the lookup
diagram shows.

Therefore, your statement here:

the anonymous superclass that
proxies the module (ASCTPTM…hehe)
does not proxy the singleton methods of the module.

is non-sensical because a regular class (or an anonymous class) never
contains the singleton methods.

When you include a module, ruby creates an anonymous
class containing the module’s instance methods, which are defined like
this:

module M
def eat
puts ‘munch, munch’
end
end

The question is what happens to the singleton methods in the module,
which are defined like this:

module M
M.greet
puts ‘MMMM’
end
end

?? Does ruby create a second class #M to contain the singleton methods?
Let’s test that:

module M
def M.greet
puts ‘MMMM’
end

def test
M.greet
end
end

class Dog
include M
end

d = Dog.new
d.test

–output:–
MMMM

So ruby does create #M because you can call the singleton methods.
However, ruby does not insert #M into the singleton hierarchy in the
lookup diagram.

This stuff isn’t easy to understand, but you can make some sense of
things. However, I find the following totally unhelpful:

Care must be taken not to confuse inheritance hierarchies: there are
two of them as 7stud has shown with the ASCII graphic. Including a
module in a class inserts it into its chain of ancestors, but has no
effect on the inheritance hierarchy of the class’s singleton class:

Basic class:

irb(main):001:0> class C;end
=> nil
irb(main):002:0> C.ancestors
=> [C, Object, Kernel, BasicObject]
irb(main):003:0> sc = class <<C;self;end
=> #Class:C
irb(main):004:0> sc.ancestors
=> [Class, Module, Object, Kernel, BasicObject]

Note that sc.ancestors does not list any of the singleton classes as
detailed in the lookup diagram.

With module included:

irb(main):005:0> module M;end
=> nil
irb(main):006:0> class C;include M;end
=> C
irb(main):007:0> C.ancestors
=> [C, M, Object, Kernel, BasicObject]
irb(main):008:0> sc.ancestors
=> [Class, Module, Object, Kernel, BasicObject]

So how does that result prove that including a module does nothing to
the lookup diagram for the singleton class? sc.ancestors never listed
the singleton classes in
the first place, and after including the module sc.ancestors still
doesn’t list the singleton classes.

12

luke gruber wrote in post #1017907:

Another, sort of related thing:

Speaking of method lookup, class objects that have singleton classes
embedded in singleton classes look up methods in the superclass’s
singleton class’s singleton class (if it exists) and actually stop at
the top-most singleton class of the original class object. A mouthful…
I find this really cool if not pretty useless :slight_smile:

I used that feature once in a solution I posted here. :slight_smile:
The op wanted to keep a method out of some other object’s
lookup path, or some such thing.

And like if a class
object has 10 singleton classes with a greet() method defined in the
seventh and eighth, and the subclass calls greet() in its sixth
singleton class, it won’t resolve.

Right. A singleton class is just an object. And all objects have a
parent class and a singleton class. But the singleton class of a
singleton class is not in the first singleton class’s lookup path. Here
is an example:

class Dog
def Dog.greet
puts ‘hello’
end
end

obj = Dog.singleton_class

obj.singleton_class.class_eval do
def speak
puts ‘woof’
end
end

Dog.greet
Dog.speak

–output:–
hello
ruby.rb:16:in <main>': undefined methodspeak’ for Dog:Class
(NoMethodError)

But:

Dog.singleton_class.speak #=> woof

Or equivalently:

class Dog
class << self
def greet
puts ‘hello’
end

class << self
  def speak
    puts 'woof'
  end
end

end

end

Dog.greet
Dog.speak

–output:–
hello
ruby.rb:18:in <main>': undefined methodspeak’ for Dog:Class
(NoMethodError)

But:

Dog.singleton_class.speak #=> woof

13

By the way, when I say that the singleton methods are in another class,
that isn’t just speculation:

class Dog
def Dog.greet
puts ‘hi’
end

def eat
puts ‘chomp, chomp’
end
end

puts Dog
puts Dog.class
puts “-” * 20
puts Dog.singleton_class
puts Dog.singleton_class.class

–output:–
Dog
Class

#Class:Dog
Class

14

Class is also an Module wich is also an Object (and there the circle is
closed)

  -------------Module
  |              ^
  |              |
  |            Class
  |             ^ ^
  |            /   \
  |          /       \
  v        /           \

Object -> #Object ##Object
^ ^ ^
| | |
Cat --> #Cat ##Cat
^ ^ ^
| | |
Dog --> #Dog ##Dog
^
|
d => #d

15

I wondered if the singleton class of a singleton class inherits from the
singleton class of a parent’s singleton class:

class Cat
class << self #Cat’s singleton class
class << self #singleton class of Cat’s singleton class
def speak
puts ‘meow’
end
end
end
end

class Dog < Cat
class << self #Dog’s singleton class
def greet
puts ‘hello’
end

end

end

Dog.greet
Dog.singleton_class.speak

–output:–
hello
meow

So the lookup diagram looks something like this:

Object -> #Object ##Object
^ ^ ^
| | |
Cat --> #Cat ##Cat
^ ^ ^
| | |
Dog --> #Dog ##Dog
^
|
d => #d

Now, we know that every class object inherits from Class. That means
any instance methods defined in class can be called by a class object,
e.g. new(). Therefore, #Object should be able to call instance methods
defined in Class:

        Class
          ^
          |

Object -> #Object ##Object
^ ^ ^
| | |
Cat --> #Cat ##Cat
^ ^ ^
| | |
Dog --> #Dog ##Dog
^
|
d => #d

But ##Object is also a class object, so it should be able to call
instance methods defined in Class as well:

        Class        Class
          ^            ^
          |            |

Object -> #Object ##Object
^ ^ ^
| | |
Cat --> #Cat ##Cat
^ ^ ^
| | |
Dog --> #Dog ##Dog
^
|
d => #d

Let’s test that hierarchy:

class Class
def eat
puts “chomp, chomp”
end
end

class Cat
class << self #Cat’s singleton class
class << self #singleton class of Cat’s singleton class
def speak
puts ‘meow’
end
end
end
end

class Dog < Cat
class << self #Dog’s singleton class
def greet
puts ‘hello’
end
end

end

Dog.greet #=>hello
Dog.singleton_class.speak #=>meow
Dog.singleton_class.eat #=>chomp, chomp
Dog.singleton_class.singleton_class.eat #=>chomp, chomp

So the lookup diagram looks like this:

               Class
                ^ ^
               /   \
             /       \
           /           \

Object -> #Object ##Object
^ ^ ^
| | |
Cat --> #Cat ##Cat
^ ^ ^
| | |
Dog --> #Dog ##Dog
^
|
d => #d

16

Therefore, your statement here:

—mine

the anonymous superclass that
proxies the module (ASCTPTM…hehe)
does not proxy the singleton methods of the module.
—mine (Luke)

is non-sensical because a regular class (or an anonymous class) never
contains the singleton methods.

7stud

Right, what I should have said, and what I really meant (that’s my story
and I’m sticking to it…) is that the … ASCTPTM … has a singleton
class, and that singleton class doesn’t proxy the included module’s
singleton class for some unknown reason. Maybe it’s because anonymous
classes can’t have singleton classes after all. Who knows about the
implementation, but that’s probably the easiest way for me to
remember/deal with it.

Anyway, thanks for the thoughts.

Now if I could only figure out how to edit the name of this thread…

-Luke

17

hehe that diagram is great. Looks like a game of hangman gone horribly
wrong.

Class is also an Module wich is also an Object (and there the circle is
closed)

Hans

But the circle doesn’t really close, as Object is an instance of Class,
and so is Module, so there is some weird looping going around.

Also, on the left-hand side of the diagram, Object looks up methods in
Class (because Object is an instance of Class), and Class inherits from
Module, which inherits from Object. So effectively, all class objects
inherit from Object twice (but obviously that doesn’t happen, so there’s
some weird exception or workaround implemented I’m sure).

-Luke

18

luke gruber wrote in post #1018148:

Therefore, your statement here:

—mine

the anonymous superclass that
proxies the module (ASCTPTM…hehe)
does not proxy the singleton methods of the module.
—mine (Luke)

is non-sensical because a regular class (or an anonymous class) never
contains the singleton methods.

7stud

Right, what I should have said, and what I really meant (that’s my story
and I’m sticking to it…) is that the … ASCTPTM … has a singleton
class, and that singleton class doesn’t proxy the included module’s
singleton class for some unknown reason. Maybe it’s because anonymous
classes can’t have singleton classes after all. Who knows about the
implementation,

Up higher somewhere, I posted this example:

====
The question is what happens to the singleton methods in the module,
which are defined like this:

module M
M.greet
puts ‘MMMM’
end
end

?? Does ruby create a second class #M to contain the singleton methods?
Let’s test that:

module M
def M.greet
puts ‘MMMM’
end

def test
M.greet
end
end

class Dog
include M
end

d = Dog.new
d.test

–output:–
MMMM

So ruby does create #M because you can call M’s singleton methods.
However, ruby does not insert #M into the singleton hierarchy in the
lookup diagram.