Superclass of eigenclass

[dannyocuiv]

On May 13, 2009, at 6:10 PM, David Flanagan wrote:

into this, I think.

I’d have to confess that this seems like a conceptual mess to me. I
presume that I’m grasping the wrong end of some stick or other. Can
anyone help point out what that might be?

It is a conceptual mess. It seems somewhat clearer, however, with
Ruby 1.9.

I agree completely. In fact, beyond implementation detail, is there
any reason that eigenclasses exist in their semi-privileged realm? For
example, I’ve never completely understood why the following shouldn’t
work:

class Test
def eigenclass
class<<self
self
end
end
end
=> nil

TestSubEigen = Class.new(Test.new.eigenclass)
TypeError: can’t make subclass of virtual class
from (irb):43:in initialize' from (irb):43:innew’
from (irb):43

In fact, it seems like the existence of eigenclasses is, to begin
with, an implementation detail. As it turns out, they can be very
handy for implementing instance-specific methods, but then isn’t that
the point of a Prototype inheritance model?

Personally, I would like to see Ruby 2.0 formalize Ruby’s prototyping
abilities. The semantics of “Test.new” could remain the same and be
conceptually identical to “Test.prototype.prototype”. That is,
eigenclasses would be prototype objects that sit between the object
and its parent class object. This would have the added benefit of
clearing up the method call chain concept, as it would simply follow
the trail of prototypes.

Just ideas…

Cheers,

Josh

[dannyocuiv]

Robert D. wrote:

However, although I agree that those terms are somehow not defined up
to the last detail in Ruby, and I have occasionaly explained about it,

You mean “complained”, right? Yes, it is kind of unfortunate, that we
ended up with this confusion. I would like to point out, however, that
this is basically a historical accident, and not anybody’s (certainly
not matz’s) fault.

The fundamental problem is that matz has always refused (and still
refuses) to pick a name. And over time he has done that for two
different reasons. Originally, he considered singleton classes to be a
private implementation detail of MRI, and he envisioned that there
would be other implementations that would choose to implement
singleton methods differently. He wanted to have per-object methods,
and he needed something to store them in, so he chose something that
was already supported by MRI: a class. Additionally, this meant that
he needed exactly 0 changes to the method resolution algorithm, and
only one single if-condition in the superclass algorithm. And he
could reuse that same solution for mixins.

Unfortunately, that supposedly private implementation was leaked to
the programmer, exactly because the singleton class is “just a
class”. Due to the way class bodies work (namely that they return the
value of the last expression inside the body instead of the class
object, and that self is bound to the class object), you can get
access to the singleton class with the ‘class << self; self end’ trick
(or its equivalent).

As soon as you can get access to something, you need to name it. And
people did. They named it singleton class, eigenclass, virtual class,
hidden class, metaclass and dozens more …

Which brings us to the present. Now matz refuses to pick a name,
because he doesn’t want to … well, pick a name. He wants the
community to pick a name, and he will simply approve whatever we
come up with. What it boils down to is that he does not consider
himself a “benevolent dictator”, but more a “benevolent chairman” who
does not himself make decisions, but rather tries to “milk” a
consensus out of a very diverse group of very opinionated people, just
like a chairman of a senate committee would (well, at least in an
ideal world where neither the members nor the chairman are corrupt or
in any way influenced by lobbyists).

Currently, singleton class, eigenclass and metaclass are still in the
race. I agree that metaclass is wrong, but since it is the name used
by _why the lucky stiff in what is probably one of the best
introductory texts about the more esoteric corners of Ruby’s object
system, a lot of Rubyists learn that name, so it is going to stick
with us. Of the other two I prefer eigenclass, both to avoid the
confusion with the singleton pattern and because it nicely fits in
with the already existing usages in mathematics (eigenvalue,
eigenvector, eigenspace, eigenmode, eigenfrequency, eigenstate,
eigenspectrum, eigenfunction, eigendecomposition, eigenproblem) and
computer science (eigenface, eigenratio).

jwm

[dannyocuiv]

On May 14, 2009, at 8:25 PM, Jörg W Mittag wrote:

also implement mixins the same way. That’s a lot of leverage for
essentially one line of code.

AFAIK, the fact that they were exposed to the programmer is more of an
accident than a well-designed thought-out language feature.

Right. So then the question: is there a compelling reason that they
should be allowed to be subclassed? (I mean, other than the obvious:
“Because that’s not the way Ruby was designed” answer.) I cannot come
up with any compelling technical reason why the artificial limitation
of being unable to subclass “virtual classes”[1] should exist. I also
don’t see what, besides this artificial limitation, is keeping Ruby
from having, at its heart, a fully prototype based inheritance system.

• Josh

[1] That’s what the exception calls them, so I’d throw that into your
pot of valid names in the running.

[dannyocuiv]

Joshua B. wrote:

In fact, it seems like the existence of eigenclasses is, to begin
with, an implementation detail.

Originally, they were.

As it turns out, they can be very
handy for implementing instance-specific methods,

Exactly. It is my understanding that they were only added as a place
to “stick” per-object methods. MRI already had classes, so why not use
them? If you stick per-object methods in an eigenclass you need
exactly 0 new constructs in the interpreter, 0 changes to method
lookup and just one if-condition in superclass lookup. Plus, you can
also implement mixins the same way. That’s a lot of leverage for
essentially one line of code.

AFAIK, the fact that they were exposed to the programmer is more of an
accident than a well-designed thought-out language feature.

jwm

[dannyocuiv]

On May 15, 2009, at 7:46 AM, Joshua B. wrote:

Right. So then the question: is there a compelling reason that they
should be allowed to be subclassed? (I mean, other than the
obvious: “Because that’s not the way Ruby was designed” answer.) I
cannot come up with any compelling technical reason why the
artificial limitation of being unable to subclass “virtual
classes”[1] should exist.

The purpose of an eigenclass is to hold per object methods. As such,
there is always always a one-to-one relationship between an eigenclass
and its “eigeninstance” (sorry, but I can’t think of a clearer term at
the moment).

The idea of subclassing an eigenclass contradicts that one-to-one
relationship. If you could subclass an eigenclass, there would be no
way to cause ruby to search the new subclass for instance methods
since it wouldn’t have a matching eigeninstance (sorry, again). It
would not be in the method lookup chain of any object.

Just to be complete, I’ll point out that there is a way for
eigenclasses to have a subclass and that is when the eigeninstances
are themselves classes. This is most clear in 1.9. If B is a
subclass of A, then B.eigenclass will be a subclass of A.eigenclass.
Even in this case A and B have to exist before their eigenclasses can
be accessed and not the other way around.

Gary W.

[dannyocuiv]

2009/5/16 Jörg W Mittag [email protected]:
But a full-fledged prototype system

would be pretty cool.
But there are! There is Ara’s prototype, which needs a little fix if I
am not mistaken or your humble servant’s one:
Smalltalking about Ruby: No More Instance Variables in My Prototypes.
Sorry for the blog link, but I have not yet repackaged PPP into Lab419
:(.
HTH
Robert

[dannyocuiv]

Joshua B. wrote:

should be allowed to be subclassed? (I mean, other than the obvious:
“Because that’s not the way Ruby was designed” answer.) I cannot come
up with any compelling technical reason why the artificial limitation
of being unable to subclass “virtual classes”[1] should exist. I also
don’t see what, besides this artificial limitation, is keeping Ruby
from having, at its heart, a fully prototype based inheritance system.

There are lots of areas in Ruby, where things aren’t as simple,
orthogonal or consistent as they should be. Unfortunately[1],
simplicity, orthogonality or consistency are not in any way design
goals. I really like your idea, but you need to find a more compelling
argument than “it makes eigenclasses consistent with classes and
simplifies the object system”. But a full-fledged prototype system
would be pretty cool.

jwm

[1] Or fortunately? After all, Brainfuck is certainly simple,
orthogonal and consistent …

[dannyocuiv]

On Sat, May 16, 2009 at 1:24 AM, Gary W. [email protected] wrote:

is always always a one-to-one relationship between an eigenclass and its
This is most clear in 1.9. If B is a subclass of A, then B.eigenclass will
be a subclass of A.eigenclass. Even in this case A and B have to exist
before their eigenclasses can be accessed and not the other way around.

And there, IMHO, is where things break down. If the definition of an
eigenclass is a ‘class’ which holds per object methods, then the class
of a class which has subclasses is not an eigenclass since it provides
methods not only to it’s sole instance but also to it’s instance’s
subclass(es).

That’s why I prefer to think of the objects which serve as the class
of a class as metaclasses, which is in line with the terminology used
in several other languages.

This whole area of Ruby reminds me a bit of Plato’s cave, we see
images on the wall cast by “smoke” and “mirrors” in the
implementation, and try to name them. And the smoke has been shifting
a bit and the mirrors have been moving slightly. When you step outside
of the cave and look at the implementation you find:

There’s a flag called FL_SINGLETON which when set in the header of a
class object means:

• It will get cloned if you clone its instance
• You can’t copy it.
  * You can’t make a subclass of it, unless you are ruby itself
  creating a subclass in which case
  subclasses of the “singleton” metaclass are created under the
  covers.
• It will be bypassed by the Object#class method
• It will be filtered out of the results Module#ancestors method
• It will be skipped in the search for methods by
  Module#instance_methods and it’s related methods
• Its methods are considered singleton methods
• It has a hidden ‘class’ instance variable attached which
  points to it’s sole instance
• Its skipped over by ObjectSpace.each_object, so
  ObjectSpace.each_object(Class) won’t yield it.
• It will prevent dumping by Marshall.dump

So what exactly does the singleton in FL_SINGLETON mean. Well, it
does mean that a class marked with FL_SINGLETON has only one direct
instance. It doesn’t mean in the case of a singleton class which
happens to be a metaclass that it doesn’t have subclasses, just that
you as a Ruby programmer can only create or add to its subclasses by
subclassing its attached class.

And while you might think that singleton method was a synonym for
instance specific method, once again this is only true when it’s a
singleton method within an instance-specific singleton class.

class Foo
def self.blat
:blat
end
end

class Bar < Foo
end

Foo.singleton_methods => [:blat]
Bar.singleton_methods => [:blat]

The 2nd ed of the Pickaxe used the term “virtual class” and seems to
call the FL_SINGLETON bit the ‘V’ bit. Looking at Ruby 1.8.6, the
term virtual class only appears in errors raised when you try to

1. Try to make a subclass of a class marked with FL_SINGLETON and
   you’ll get the error
   “can’t make subclass of virtual class”
2. Try to add a singleton method to an instance of an immediate
   class (e.g. a Fixnum), since there’s no klass ptr so nowhere to insert
   the needed instance specific class in the lookup chain, and you’ll get
   the error “no virtual class for %s”
3. Try to instantiate an instance of a class marked with
   FL_SINGLETON and you’ll get the error can’t create instance of virtual
   class"

And in Ruby 1.9, these error descriptions have all be changed to
remove the usage of “virtual class” and replace it with “singleton
class”.

I kind of wish that FL_SINGLETON had actually been named FL_VIRTUAL,
or FL_HIDDEN, since what the flag does mostly is to hide such classes
in the standard library provided API. (ignoring the leak from class <
a;self;end

Back in the mists of Ruby history, it became popular to call any class
marked with FL_SINGLETON as a metaclass, probably because that’s what
_why named the leak method in the Poignant Guide.

That added another layer of confusion, because to many of us metaclass
means the class of a class, and an instance-specific class ain’t a
metaclass, it’s a class whether it’s hidden or not.

And I think that that’s where the name eigenclass got coined, as an
alternative to singleton class, for better or worse. And with the
publication of “The Ruby P.ming Language” it’s gotten a bit of a
blessing from Matz, although the Ruby 1.9 implementation uses
singleton class in those errors where virtual class was used before.

I personally think that the distinction between metaclasses (in the
generally accepted sense of the class of a class) and
instance-specific classes is more important than the accidental
similarities between singleton classes used as instance-specific
classes and singleton-classes used as metaclasses.

And I guess eigenclass is not a bad term to use for a class which
provides behavior to a single instance, i.e. an instance-specific
class, but I don’t like it as a term for a metaclass.

But that’s just me, after all it’s all just images caused by smoke and
mirrors

–
Rick DeNatale

Blog: http://talklikeaduck.denhaven2.com/
Twitter: http://twitter.com/RickDeNatale
WWR: http://www.workingwithrails.com/person/9021-rick-denatale
LinkedIn: http://www.linkedin.com/in/rickdenatale

[dannyocuiv]

Danny O cuiv wrote:

There seems to be two forces contributing to this. Firstly, the
profusion of terms (many of which are clearly misnomers) is getting out
of hand, and is unhelpful to newcomers. Secondly, many people still
resort to referring to MRI runtime structures when discussing semantics
even though Ruby is now being implemented on the JVM, on the CLR, on the
Objective-C runtime and presumably elsewhere.

Yet more runtimes: the Parrot VM (Cardinal), a modified version of the
64-Bit Gemstone Smalltalk VM (MagLev), the Flash/ActionScript Tamarin
VM (Red Sun) and the SAP NetWeaver/ABAP VM (Blue Ruby). Plus
specialized Ruby VMs (Rubinius, tinyrb).

There is now a clear need
to be able to discuss semantics at a sufficient level of abstraction,
i.e. purely in terms of what is revealed by the reflection API. If there
are any inconsistencies or anomalies revealed by the API, then this
should be addressed directly.

Let’s not forget the current ISO standardization work for Ruby.
Anybody remember Open OfficeXML? It was standardized by
reverse-engineering MS Word, exactly the way that you describe (which
is basically the way RubySpec works: reverse-engineering MRI and
YARV). In the end, it was so incredibly hard to implement that even
Microsoft itself shipped a working implementation of ODF (i.e. the
native OpenOffice.Org XML format) before OOXML (i.e. the native MS
Office XML format). [In fact, OOXML is widely believed to be
un-implementable, because of logical contradictions in the spec.]

We need a specification of Ruby’s semantics, independent of MRI
and/or YARV, both for the sake of all the other implementations and
ISO Ruby.

Unfortunately, there is really only one person that could write such a
spec.

Ideally, of course, that specification would be both formal and
abstract enough, so that we can automatically generate an interpreter
from it, or at least a testsuite. Or feed it to an automatic theorem
prover, and have it prove interesting properties like memory safety,
$SAFE-level safety, consistency etc.

And now, I’m going down to the stables to feed my unicorn

The central point that I think should be made here is that per-instance
behaviour has nothing to do with inheritance, per se. One obvious
language rule will be that per-instance behaviour should take precedence
over instance-common behaviour and one convenient way of implementing
that is to snap an eigenclass into the chain between the instance and
its class. But that implementation-specific runtime structure won’t
serve as a definition of the semantics for implementations that work
differently. Take, for example, a “Ruby in the browser” (perhaps one
already exists?) that translates onto the JavaScript object model.

HotRuby was (is?) an implementation of not “Ruby in the browser” but
“YARV in the browser”. It was basically a YARV bytecode interpreter
loop written in JavaScript (meaning it requires that the Ruby
sourcecode gets pre-compiled with the actual YARV compiler, the
bytecode dumped to disk and then delivered to the browser as a binary
blob).

YARI (Yet Another Ruby Interpreter) aka IoRuby was an implementation
of Ruby (1.6, I think – it was before my time) on the Io runtime in
Io, which is a prototype-based OO language.

There has been talk about implementing Ruby on top of V8, which is an
ECMAScript engine.

There has been talk about implementing Ruby on top of NekoVM, which
has a unified class/prototype model.

None of these implementations actually have come even close to doing
anything even remotely useful and of course couldn’t even dream of
running (let alone passing) RubySpec. But, there is at least some
interest in running Ruby on top of “prototype-ish” runtimes.

jwm

[dannyocuiv]

Rick Denatale wrote:

But that’s just me, after all it’s all just images caused by smoke and mirrors

Perhaps. But there’s also the more mundane possibility that it’s
just inconsistent semantics and bad terminology.

There seems to be two forces contributing to this. Firstly, the
profusion of terms (many of which are clearly misnomers) is getting out
of hand, and is unhelpful to newcomers. Secondly, many people still
resort to referring to MRI runtime structures when discussing semantics
even though Ruby is now being implemented on the JVM, on the CLR, on the
Objective-C runtime and presumably elsewhere. There is now a clear need
to be able to discuss semantics at a sufficient level of abstraction,
i.e. purely in terms of what is revealed by the reflection API. If there
are any inconsistencies or anomalies revealed by the API, then this
should be addressed directly.

The central point that I think should be made here is that per-instance
behaviour has nothing to do with inheritance, per se. One obvious
language rule will be that per-instance behaviour should take precedence
over instance-common behaviour and one convenient way of implementing
that is to snap an eigenclass into the chain between the instance and
its class. But that implementation-specific runtime structure won’t
serve as a definition of the semantics for implementations that work
differently. Take, for example, a “Ruby in the browser” (perhaps one
already exists?) that translates onto the JavaScript object model. It
would make perfect sense for this to implement per-instance behaviour as
object properties. The class << obj syntax would then just be a way of
manipulating the per-instance behaviour collectively. The only
assumption being that method resolution is correctly (and hopefully
efficiently) implemented. And the eigenclass would be purely a
conceptual notion, existing only as several properties of an object,
taken together. No problem whatsoever.

It seems that the snapping of eigenclasses into the structure was
intended to be a private implementation detail. Certain reflective
methods were written in such a way as to skip over the eigenclasses, as
though they were not to be regarded as part of the public (semantic)
inheritance hierarchy. However, there were one or two leaks. For
example, eigenclass.superclass reports the original class of the
instance (in 1.9 at least). And, in a similar vein, the keyword super
chains from an instance-specific method to a regular instance method.
(As it happens, it transpires that this is convenient way of enabling
interception.) Ideally, one would like to stem the leaks as far as
possible without causing too much breakage of existing code.

Here’s a few conclusions I’ve come to. I’d be interested to hear what
others think.

(i) The correct way to present method resolution is the way
Flanagan/Matz do on page 258 of their book, i.e. as an algorithm. It’s
simple, intuitive and easy to remember. Ruby implementations are free to
implement this algorithm any way that’s convenient (and fast) in their
given context. The wrong way to present method resolution is the Pickaxe
approach of using diagrams from MRI’s runtime structure. Apart from not
being portable (which will become a bigger and bigger problem), it’s
backwards. That runtime structure exists like that in order to support
certain desired semantics. So deriving the semantics from the structure
is neither here nor there. Just state them up front.

(ii) The keyword super should just be defined to work according to the
method resolution algorithm, not the inheritance hierarchy. This is what
happens at the moment, so it’s just a formalising of the de facto
situation and doesn’t break anything. This would preserves the
convenient chaining from a per-instance method to the regular instance
method and makes it clear that there are potentially lots of other
useful chaining opportunities. As Gary pointed out to David earlier in
this thread, the sequence of classes produced by the method resolution
algorithm is different from the inheritance hierarchy because of module
mixin and eigenclasses. (And in Ruby, module mixin is not inheritance,
regardless of the similarity.)

(iii) The superclass method should be defined to return Object for the
eigenclass of a regular object. In other words, the original statement I
alluded to from page 261 of RPL would now be true. The current situation
(returning the class of the instance) is an implementation leak. I don’t
think there would be much breakage resulting from this and it cleans
things up.

(iv) In principle, class method resolution could also be defined purely
by an algorithm, as a language rule, which would then not require
explicit semantic inheritance to work.

Lastly, the terminology situation needs to be addressed by the
community. The term eigenclass is really spot on for a conceptual
container for per-instance behaviour and it’s great that RPL adopted it.
The other terms are not acceptable synonyms; they are misnomers.

(a) metaclass
Even _why, in the 2005 piece that led to the widespread use of the term
metaclass, acknowledged that the term was wrong: ‘This definition
doesn’t really work with Ruby, though, since “a class which defines a
class” is simply: a Class’. The simplest way to illustrate this is to
compare Smalltalk and Ruby. In Smalltalk, if you evaluate the expression
“Integer class”, you get “Integer class” (i.e. the same thing). The
answer being the same as the question is an expression of the anonymous
nature of the metaclass, but is read by a Smalltalk programmer as “the
metaclass of Integer”. In Ruby, if you evaluate the expression
“Integer.class”, you get “Class”. Thus, even if the eigenclasses of
class objects in Ruby play a similar role to metaclasses in Smalltalk,
they are not metaclasses. By definition.

(b) singleton class
As pointed out in the “Ruby in the browser” example above, there is no
need for the eigenclass to actually exist as a regular class, let alone
have a single instance. Clearly then, the term singleton class would
only be appropriate to some implementations and would consequently be a
misnomer.

While I’m at it, the term “singleton method” is also a misnomer. The
term “singleton” derives from set theory. A singleton set is a set with
one element. As such, the GoF pattern is a perfect fit. However, there
is no singleton set anywhere in this situation (unless you get really
silly in your choice of set). You can have many instances, each with
many per-instance methods. The only appearance of the word “one” is on
one side of the cardinality of an association (and not on the method
side). For reasons of symmetry, eigenmethod is a far better term.

Regards,

Danny.