A C extension Gem

Ruby JRuby
1

Hey guys,

Firstly I know that having a C extension in JRUBY is not a great idea,
however, I am kind of being forced in a way.

I have a system at the moment that uses a Ruby KDTree

It works, but its not very quick. I am using Jruby because of its great
performance and would love to stay with it.

To improve the speed of the KDTree, I found and used
GitHub - consti/tupalo-kdtree: Threadsafe KD-Tree Gem which is crazy quick, it also
has
the same syntax for use
http://rubydoc.info/gems/tupalo-kdtree/0.2.3/framesand it works great
under MRI 1.9.3 - it brings down on benchmark from 30
seconds to around 0.01 seconds.

However when I try and load this gem in Jruby it only works to load
using
JRUBY_OPTS=–1.8 - which is odd, but it loads find and installs.

However when I run the attached code, it runs incredibly slow. unless
you
decrease the number of objects going into the KDTree - almost as if it
runs
out of memory?

require ‘rubygems’
require “benchmark”

gem ‘tupalo-kdtree’
require ‘kdtree’

#set the number of lat/lngs to go into KDTree
@kdsize = 10000000

sets up random lat longs and loads them into the KDTree - len = number

of
lat/lngs
def setup_tree(len)
@points = (0…len).map { |i| [rand_coord, rand_coord, i ] }
@kdtree = KDTree.new(@points)
end

little method to generate random lat/lngs

def rand_coord
rand(0) * 10 - 5
end

looks up random lat long to find nearest neighbour inside KDTree

def test_nearestk
pt = []
@list = []
2000.times do
pt = [rand_coord, rand_coord]
@list << @kdtree.nearest(pt[0],pt[1])
end
end

#create KDTree (takes the longest)
setup_tree(@kdsize)

#Benchmark the time it takes to lookup @list for nearest neighboard in
@kdtree
Benchmark.bm do |x|
x.report do
test_nearestk
end
end

If you change @kdsize to 1000 it works pretty well!

I do not know why this doesn’t run will inside of Jruby, and if it is a
really stupid idea can someone think of an alternative? I am a bit
stuck!
But would love to get away from the 30 seconds it takes to run the
original
ruby code and use the C code.

This is my first post in here, sorry if it is too much!

Thanks Charlie

2

We talked a bit on IRC, but I’ll recap here.

JRuby 1.6.x does support C extensions in an experimental way, but it’s
strongly discouraged for a few reasons:

  • C exts on managed runtimes like JRuby will always have performance
    issues compared to MRI.
  • C exts crippled concurrency on all runtimes, since we can’t run
    their code in parallel.
  • There’s nobody on the JRuby team who will update JRuby’s C ext
    support for 1.9 mode.

It’s also likely that JRuby 1.7 will ship with C exts disabled by
default, since they tend to cause more problems than they solved.

That said, your best courses of action are:

  • Try the Ruby version on JRuby 1.7 (master) on Java 7; it may be
    significantly faster.
  • Work with us to improve the performance of the Ruby version.
  • If the Ruby version won’t cut it, look for a Java library that does
    what you need and call it from Ruby code.
  • If there’s no Java version, port the C code (or hire/beg someone to
    do it for you)
  • Charlie
3

I made some changes to the C ext support in jruby-master that might
make that tupalo-kdtree gem not quite as slow. You will have to
recompile jruby, the cext layer (‘ant cext jar’ should do it) , and
reinstall the tupalo-kdtree gem.

Background (and why C exts are bad on anything that is not MRI):
whenever an array is accessed via RARRAY() in jruby, the jruby cext
layer cannot just present a pointer to the array contents like MRI
does. It must copy and convert the ruby array to a C representation,
then sync the contents of the C and ruby array on each transition
to/from C code.

RARRAY_LEN() was naively using RARRAY(), which caused the shadow C
array to be setup, with all the syncing, and then kdtree was using
rb_ary_entry() to pull out each element of the array … and since
rb_ary_entry() causes a transition between C and ruby, it was causing
the array to be synced each way. So, you end up with something like
O(2 * 3 * n^2), and when you throw an array of length 10 million at
that, you iterate over 10000000 * 10000000 * 6 elements, copying them
to/from ruby. Thats … gonna take a while.

4

Charlie (great name btw)

Thanks for getting back to me, it is greatly appreciated.

I understand fully about the C issue, thanks for going into it for me.
After some testing on Jruby-Head with Java 7 there were massive speed
improvements on the Ruby Code, went from 30 to 12 seconds. Which is
amazing

Due to a project need I have hired someone to code the the gem into
java. I
looked into using a Java library instead but we had already written a
lot
of code that uses the gem, and would have required more of a rewrite
this
end.

I am more than happy to help in testing Jruby further, or working with
you
to improve the ruby code I had.

We are also successfully running jRuby in a production environment, I
saw a
tweet from you guys a while back saying you might be looking for blog
posts
from companies that are running Jruby. Are you still interested?

Once again thanks for getting back to, and thanks for all your great
work!

Charlie

5

Okay, wow that is pretty good of you, I will try it out, and thanks for
the
explanation

C

6

Take a look at Weka Weka 3 - Data Mining with Open Source Machine Learning Software in Java. It has a Java
KD
tree implementation
KDTree
.