IronRuby FFI

Ruby IronRuby
1

I know that we’ve discussed this in the past, but I’m interested in
doing it
for two reasons:

  1. We use mono with a bridge to ObjectiveC and Cocoa, and we’d like to
    investigate libffi via mono as a potential replacement for our current
    bridge.
  2. I’m interested just for the sake of learning more about FFI.

Mono appears to have had a libffi implementation that was later removed,
so
I think I have a place to start. However, I’m not sure that’s the right
starting point. Does anyone have a suggestion for how to get started?
I’ve
been taking a look at libffi and DllImport, but I’m not sure if those
are
the right directions, something else, or what.

Thanks,

Ryan R.

Email: [email protected]
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/

2

Ryan,

I’m right there with you, only I looked at JFFI for inspiration (didn’t
know
mono had anything - could you share more about that?). In fact, In my
infinite laziness, I posted a job for a couple hundred bucks on
Rent-A-Coder, hoping someone could essentially port JFFI to C#, so I
could
focus on writing the actually IronRuby library… but nothing came of
that.

I’m tempted to suck it up and start coding this myself. Would you be
interested in working together? I figured I’d take the approach of
essentially writing “NFFI”, and then write an IronRuby lib around that.

-Charles

3

Couldn’t this be implemented via P/Invoke? Would P/Invoke work without
the
attributes required in C#?

–
Will G.
http://hotgazpacho.org/

On Fri, Aug 20, 2010 at 4:49 PM, Charles S. <

4

That’s something I’m curious about too. I don’t know whether or not
P/Invoke
alone can match the functionality of libffi. I’d have to look through
some
of libffi and see if there’s anything can’t be accomplished in C#.

If everything can be accomplished in C#, I’d go that route. We could
emit
types at runtime that have the necessary DLLImport attributes and such
(something I’ve already done for little projects of mine).

-Charles

5

I’ll have to take a look at that. If it’s cross platform, it could be a
great resource.

-Charles

On Fri, Aug 20, 2010 at 4:57 PM, Tomas M. <

6

Charles, I’m happy to work with you to get this done. I’m getting close
to finishing some projects and will have more time to work on it in a
few weeks. I will send the info I got from the mono-devel list.
Where/how do you want to start?

Ryan

Sent from my iPhone

7

I would also recommend to look at Python’s c-types and especially
IronPython’s implementation:

http://github.com/ironruby/ironruby/tree/master/Languages/IronPython/IronPython.Modules/_ctypes/

Many of the features might be similar to Ruby’s FFI and it would be good
to share the implementation with IronPython when possible.

Tomas

From: [email protected]
[mailto:[email protected]] On Behalf Of Charles
Strahan
Sent: Friday, August 20, 2010 2:35 PM
To: [email protected]
Subject: Re: [Ironruby-core] IronRuby FFI

That’s something I’m curious about too. I don’t know whether or not
P/Invoke alone can match the functionality of libffi. I’d have to look
through some of libffi and see if there’s anything can’t be accomplished
in C#.

If everything can be accomplished in C#, I’d go that route. We could
emit types at runtime that have the necessary DLLImport attributes and
such (something I’ve already done for little projects of mine).

-Charles
On Fri, Aug 20, 2010 at 4:14 PM, Will G.
<[email protected]mailto:[email protected]> wrote:
Couldn’t this be implemented via P/Invoke? Would P/Invoke work without
the attributes required in C#?

–
Will G.
http://hotgazpacho.org/

On Fri, Aug 20, 2010 at 4:49 PM, Charles S.
<[email protected]mailto:[email protected]> wrote:
Ryan,

I’m right there with you, only I looked at JFFI for inspiration (didn’t
know mono had anything - could you share more about that?). In fact, In
my infinite laziness, I posted a job for a couple hundred bucks on
Rent-A-Coder, hoping someone could essentially port JFFI to C#, so I
could focus on writing the actually IronRuby library… but nothing came
of that.

I’m tempted to suck it up and start coding this myself. Would you be
interested in working together? I figured I’d take the approach of
essentially writing “NFFI”, and then write an IronRuby lib around that.

-Charles

On Fri, Aug 20, 2010 at 2:33 PM, Ryan R.
<[email protected]mailto:[email protected]> wrote:
I know that we’ve discussed this in the past, but I’m interested in
doing it for two reasons:

  1. We use mono with a bridge to ObjectiveC and Cocoa, and we’d like to
    investigate libffi via mono as a potential replacement for our current
    bridge.
  2. I’m interested just for the sake of learning more about FFI.

Mono appears to have had a libffi implementation that was later removed,
so I think I have a place to start. However, I’m not sure that’s the
right starting point. Does anyone have a suggestion for how to get
started? I’ve been taking a look at libffi and DllImport, but I’m not
sure if those are the right directions, something else, or what.

Thanks,

Ryan R.

Email: [email protected]mailto:[email protected]
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/

Ironruby-core mailing list
[email protected]mailto:[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

Ironruby-core mailing list
[email protected]mailto:[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

8

Ryan,

Sorry for the long delay - I meant to give it some thought before I got
back
to you… and know it’s been quite some time.

I think it would be a good idea to replicate JFFI, using P/Invoke
directly,
if possible (as opposed to P/Invoking libffi
http://www.cygwin.org/libffi/).
That would give us a good separation of concerns and a reusable library,
and
possibly an easy way to port any Java/JRuby code that uses JFFI to
C#/.NET
too.

I’m about to set up an NFFI repo at http://github.com/cstrahan/nffi. - I
suppose you could fork it and send me pull requests (unless you have a
better workflow in mind - I’m definitely not a git guru). I’ve been
learning
C/C++ the last couple months, so I should be able to write simple DLL to
run
our tests against. I think I’ll take a TDD approach to driving out the
C#
lib. Once we have NFFI working, it should be relatively straightforward
to
expose that to the IronRuby runtime. I’ll try to get something pushed
out to
my repo by the end of tomorrow - I’ll keep you in the loop.

That’s what I have in mind, but I’m open to suggestions.

-Charles

9

Sounds good to me!

Sent from my iPhone

10

From the mono-devel list:

On Thu, 2010-08-19 at 10:35 -0700, Ryan R. wrote:

Has anyone created or investigated Mono support for libffi?

Once upon a time, Mono used libffi directly. It was removed in r724
(git SHA1 ID d0cd6059c1b2edad12eb67cb8e64b3cd187be1b1) on 2001-09-05
(and earlier). Unfortunately, the commit message is useless, but iirc
the reason for removing it was because it was significantly slower than
what Mono could do itself.

I’d like to contribute this to help support Ruby-FFI for IronRuby.

I imagine IronRuby doesn’t require it’s own FFI, it would just use the
usual .NET FFI of DllImport, no?

As for supporting Ruby-FFI, perhaps you can extend the existing Mono FFI
support to support Ruby? I don’t know what would be involved…

  • Jon

Sent from my iPhone

11

I’m also taking a look at IronPython’s CTypes implementation, under
Tomas’
advice. I’ve noticed that their
MemoryHolderhttp://github.com/ironruby/ironruby/blob/master/Languages/IronPython/IronPython.Modules/_ctypes/MemoryHolder.csclass
derives from
CriticalFinalizerObjecthttp://msdn.microsoft.com/en-us/library/system.runtime.constrainedexecution.criticalfinalizerobject.aspx,
which led me to the discovery of Constrained Execution
Regionshttp://msdn.microsoft.com/en-us/library/ms228973.aspx
.

I sent an inquiry to the IronPython mailing
listhttp://lists.ironpython.com/pipermail/users-ironpython.com/2010-October/013757.htmlregarding
the use CFO, and about CER in general, as I haven’t had any
exposure to either, and the MSDN docs are a little daunting. If anyone
here
would like to give an explanation of either one, that would be awesome.

Any experience with either of those, Ryan?

-Charles

12

I’ve decided to not be lazy and do a little spelunking into CER’s - it’s
rather interesting stuff. I found a pretty good article here:

In laymen’s terms, it looks like CER’s provide reliability where
asynchronous exceptions may be thrown: OutOfMemoryException,
StackOverflowException, and ThreadAbortException. In the case of
MemoryHolder, this is important because such exceptions could preempt
the
storage of IntPtrs corresponding to allocated memory and/or the
deallocation
of memory within finalizers - both resulting in memory leaks. As I
imagined,
this will be something we’ll want to incorporate in our FFI impl.

http://msdn.microsoft.com/en-us/magazine/cc163716.aspx-Charles

13

I’ve not heard of any of those. I started looking at ctypes but never
got far.

Sent from my iPhone

14

Here’s Dino Viehland’s response regarding the CERs:


Objects which inherit from CriticalFinalizerObject basically have their
finalize method turned into a CER.  The finalizer method is also JITed
before any instances are created so the finalizer is guaranteed to be
runnable.



In generally CERs are all about ensuring that the VM won’t cause any
unexpected failure points in your code.  These can be introduced because 
the
VM does something lazily (including JITing methods) and doing the work 
might
fail due to insufficient resources.  Or it can also be due to thread 
abort.
Because these objects are responsible for freeing up resources we don’t 
want
any unexpected failures to be injected otherwise the resources would 
leak.



So for example MemoryHolder also has a CER in the constructor – this 
ensures
that we don’t take a ThreadAbort between the CallocCall, storing the 
value
in _data, or assigning to _ownsData.  This will all complete or not 
complete
so that our state is consistent when the finalizer is run.  It also 
makes
sure that any work the CLR needs to perform to call 
NativeFunctions.Calloc
is all performed before we enter the CER so that we don’t get an out of
memory exception while calling or returning from it.



For most environments it’s not super important that this is gotten right 
–
but if you run in a process which needs long uptime, is resource
constrained, and/or uses thread abort a lot (SQL server being an example 
of
all 3) it’s important that this is correct.  I happened to work on this
feature when I was on the CLR team so it came rather naturally to me to 
get
it right J

-Charles

On Mon, Oct 4, 2010 at 12:11 AM, Charles S. <

15

@Charles

Have you made any progress here? If so, what’s your repo? I’m looking
into
this again this morning.

Cheers,

Ryan R.

Email: [email protected]
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/

On Sun, Oct 3, 2010 at 11:47 PM, Charles S. <

16

Charles,

Have you made any progress with FFI? I have finally finished with all of
my
craziness. Of course, the holidays are approaching, but I would love to
investigate this further in the new year.

~ Ryan

From: [email protected]
[mailto:[email protected]] On Behalf Of Charles
Strahan
Sent: Sunday, October 03, 2010 11:47 PM
To: [email protected]
Subject: Re: [Ironruby-core] IronRuby FFI

Here’s Dino Viehland’s response regarding the CERs:


Objects which inherit from CriticalFinalizerObject basically have their
finalize method turned into a CER.  The finalizer method is also JITed
before any instances are created so the finalizer is guaranteed to be
runnable.



In generally CERs are all about ensuring that the VM won't cause any
unexpected failure points in your code.  These can be introduced because 
the
VM does something lazily (including JITing methods) and doing the work 
might
fail due to insufficient resources.  Or it can also be due to thread 
abort.
Because these objects are responsible for freeing up resources we don't 
want
any unexpected failures to be injected otherwise the resources would 
leak.



So for example MemoryHolder also has a CER in the constructor - this 
ensures
that we don't take a ThreadAbort between the CallocCall, storing the 
value
in _data, or assigning to _ownsData.  This will all complete or not 
complete
so that our state is consistent when the finalizer is run.  It also 
makes
sure that any work the CLR needs to perform to call 
NativeFunctions.Calloc
is all performed before we enter the CER so that we don't get an out of
memory exception while calling or returning from it.



For most environments it's not super important that this is gotten right 
-
but if you run in a process which needs long uptime, is resource
constrained, and/or uses thread abort a lot (SQL server being an example 
of
all 3) it's important that this is correct.  I happened to work on this
feature when I was on the CLR team so it came rather naturally to me to 
get
it right J

-Charles

On Mon, Oct 4, 2010 at 12:11 AM, Charles S.
[email protected] wrote:

I’ve decided to not be lazy and do a little spelunking into CER’s - it’s
rather interesting stuff. I found a pretty good article here:

In laymen’s terms, it looks like CER’s provide reliability where
asynchronous exceptions may be thrown: OutOfMemoryException,
StackOverflowException, and ThreadAbortException. In the case of
MemoryHolder, this is important because such exceptions could preempt
the
storage of IntPtrs corresponding to allocated memory and/or the
deallocation
of memory within finalizers - both resulting in memory leaks. As I
imagined,
this will be something we’ll want to incorporate in our FFI impl.

-Charles

On Sun, Oct 3, 2010 at 11:51 PM, Ryan R.
[email protected]
wrote:

I’ve not heard of any of those. I started looking at ctypes but never
got
far.

Sent from my iPhone

On Oct 3, 2010, at 9:27 PM, Charles S.
[email protected]
wrote:

I’m also taking a look at IronPython’s CTypes implementation, under
Tomas’
advice. I’ve noticed that their MemoryHolder
<http://github.com/ironruby/ironruby/blob/master/Languages/IronPython/IronPy
thon.Modules/_ctypes/MemoryHolder.cs> class derives from
CriticalFinalizerObject
<System.Runtime.ConstrainedExecution Namespace | Microsoft Learn
.criticalfinalizerobject.aspx> , which led me to the discovery of
Constrained Execution Regions
http://msdn.microsoft.com/en-us/library/ms228973.aspx .

I sent an inquiry to the IronPython mailing list
<http://lists.ironpython.com/pipermail/users-ironpython.com/2010-October/013
757.html> regarding the use CFO, and about CER in general, as I haven’t
had
any exposure to either, and the MSDN docs are a little daunting. If
anyone
here would like to give an explanation of either one, that would be
awesome.

Any experience with either of those, Ryan?

-Charles

On Sun, Oct 3, 2010 at 10:39 PM, Ryan R.
[email protected]
wrote:

Sounds good to me!

Sent from my iPhone

On Oct 3, 2010, at 8:04 PM, Charles S.
[email protected]
wrote:

Ryan,

Sorry for the long delay - I meant to give it some thought before I got
back
to you… and know it’s been quite some time.

I think it would be a good idea to replicate JFFI, using P/Invoke
directly,
if possible (as opposed to P/Invoking libffi
http://www.cygwin.org/libffi/
). That would give us a good separation of concerns and a reusable
library,
and possibly an easy way to port any Java/JRuby code that uses JFFI to
C#/.NET too.

I’m about to set up an NFFI repo at http://github.com/cstrahan/nffi. - I
suppose you could fork it and send me pull requests (unless you have a
better workflow in mind - I’m definitely not a git guru). I’ve been
learning
C/C++ the last couple months, so I should be able to write simple DLL to
run
our tests against. I think I’ll take a TDD approach to driving out the
C#
lib. Once we have NFFI working, it should be relatively straightforward
to
expose that to the IronRuby runtime. I’ll try to get something pushed
out to
my repo by the end of tomorrow - I’ll keep you in the loop.

That’s what I have in mind, but I’m open to suggestions.

-Charles

On Sat, Aug 21, 2010 at 9:23 PM, Ryan R.
[email protected]
wrote:

Charles, I’m happy to work with you to get this done. I’m getting close
to
finishing some projects and will have more time to work on it in a few
weeks. I will send the info I got from the mono-devel list. Where/how do
you
want to start?

Ryan

Sent from my iPhone

On Aug 20, 2010, at 1:49 PM, Charles S.
[email protected]
wrote:

Ryan,

I’m right there with you, only I looked at JFFI for inspiration (didn’t
know
mono had anything - could you share more about that?). In fact, In my
infinite laziness, I posted a job for a couple hundred bucks on
Rent-A-Coder, hoping someone could essentially port JFFI to C#, so I
could
focus on writing the actually IronRuby library… but nothing came of
that.

I’m tempted to suck it up and start coding this myself. Would you be
interested in working together? I figured I’d take the approach of
essentially writing “NFFI”, and then write an IronRuby lib around that.

-Charles

On Fri, Aug 20, 2010 at 2:33 PM, Ryan R.
[email protected]
wrote:

I know that we’ve discussed this in the past, but I’m interested in
doing it
for two reasons:

  1. We use mono with a bridge to ObjectiveC and Cocoa, and we’d like to
    investigate libffi via mono as a potential replacement for our current
    bridge.

  2. I’m interested just for the sake of learning more about FFI.

Mono appears to have had a libffi implementation that was later removed,
so
I think I have a place to start. However, I’m not sure that’s the right
starting point. Does anyone have a suggestion for how to get started?
I’ve
been taking a look at libffi and DllImport, but I’m not sure if those
are
the right directions, something else, or what.

Thanks,

Ryan R.

Email: [email protected]
LinkedIn: http://www.linkedin.com/in/ryanriley
Twitter: @panesofglass
Blog: http://wizardsofsmart.net/
Website: http://panesofglass.org/

Ironruby-core mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

Ironruby-core mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

Ironruby-core mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

Ironruby-core mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

Ironruby-core mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

Ironruby-core mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ironruby-core

17

Ryan,

Have you made any progress with FFI? I have finally finished with all of
my

craziness. Of course, the holidays are approaching, but I would love to
investigate this further in the new year.

I haven’t written any code yet (I’ve been rather swamped lately). I took
a
look at the API (http://rdoc.info/github/ffi/ffi) that we’d have to
provide
for a full implementation, and it looks rather daunting. I’m still
reading
through JFFI, JRuby and RubyFFI trying to wrap my head around all of it.

I’m not sure how I would get started right now. However, I’m going to
spend
some time the next couple days trying to figure out how I might attack
this.
I’ll be sure to keep this thread up-to-date with my thoughts/feelings on
this.

Feel free to take the lead if you have a clearer vision of how this
needs to
get done; otherwise, I’ll update you on Tuesday or Wednesday.

Cheers,
-Charles