Benchmarks sorta slow, am I using right options?

Ruby JRuby
1

Hi,

I just ran a bunch of JSON parsing/generating benchmarks with MRI, JRuby
1.6 and JRuby 1.7 in combination with a bunch of different JSON
libraries.

The JRuby results… well when I saw them I sorta got that look on my
face like when you’re at a party, you take a swig of beer and then you
just shit yourself. Everyone around you gets this awkward look, you
look around and avoid eye contact but they know what you did, you know
what you did… oh… you’re not relating to that one? Ok, “awkward” is
what I’m going for here. Awkward.

I ran JRuby with the following options:
–1.9 --server -Xcompile.mode=FORCE --fast

I thought with those options pitting JRuby against MRI would be like
shooting fish in a barrel (or your girlfriend’s fish tank) given how I
imagine JSON parsing/generation would work and all the magical
optimisations I imagine JRuby would be able to do in those cases.

So, should I be using different options?

Btw sorry if I have unrealistic expectations of JRuby. It IS an
awesome product and I’m really looking forward to seeing how the 1.7
tree comes along.

Finally, benchmark results here if you want a looksee:

Cheers guys.
David

2

Thanks for bringing this to the list! Comments below.

On Wed, Apr 18, 2012 at 12:52 PM, David B. [email protected]
wrote:

what you did… oh… you’re not relating to that one? Ok, “awkward” is
what I’m going for here. Awkward.

I ran JRuby with the following options:
–1.9 --server -Xcompile.mode=FORCE --fast

These options are fine, though with 1.6+ you don’t really need --fast
(almost everything interesting it does became default in 1.6).

JVM version does make a difference; Oracle/OpenJDK 7 (1.7) is faster,
and in combination with JRuby 1.7 it can be even faster for running
Ruby code.

I thought with those options pitting JRuby against MRI would be like
shooting fish in a barrel (or your girlfriend’s fish tank) given how I
imagine JSON parsing/generation would work and all the magical
optimisations I imagine JRuby would be able to do in those cases.

Comments on the individual JSON impls you tested…

YAJL:

Anything C extension-based is going to be slow on JRuby (and every
impl except MRI, for various reasons). We discourage using C exts and
our support for them in 1.6 was mostly an olive branch.

Oj:

Same statement as above, I would imagine. I’m amazed either of them
worked as well as they did :slight_smile:

json-pure:

This is pure Ruby, so it’s always going to suffer compared to native
code. However, JRuby 1.7 on Java 7 should do considerably better than
other Rubies, earlier JRubies, or earlier Javas. There are some cases
where JRuby + Java 7 results in worse performance than JRuby + Java
6; if that’s the case here, I really want to know.

json-jruby:

At this point I believe you should just use the “json” gem, which
includes the json-jruby stuff, or the “json” 1.9.3 library in JRuby
master which is the same thing.

I have looked into the performance of the json gem a few times, and
more recent builds (last few months) should have some perf
improvements I made. Ultimately the remaining performance issue comes
down to problems with the generated json code we can’t really do
anything about (since it’s generated by Ragel) and really has nothing
to do with JRuby.

I’d love to see a port of Oj (or YAJL) to Java that we could connect
with directly, but I don’t know of any plans for someone to do that.

The bottom line is that you’re really testing a library and the
integration with it rather than individual impls here. MRI + C exts
can be very fast. JRuby + C exts is slow. MRI + Ruby code can be slow.
JRuby + Ruby code can be pretty fast. JRuby + Java libraries can be
fast, if the libraries themselves are fast.

I hope that clears things up. I’d love to see the JRuby JSON options
become competitive with the fastest MRI + C options, but it would take
some work.

  • Charlie
3

Complete outsider’s possibly dumb comment: Have you considered wrapping
a fast Java JSON library such as Jackson and providing that as a gem? I
wonder how that would compare to the other implementations?

4

On Wed, Apr 18, 2012 at 9:21 PM, Tim F. [email protected] wrote:

Complete outsider’s possibly dumb comment: Have you considered wrapping a
fast Java JSON library such as Jackson and providing that as a gem? I wonder
how that would compare to the other implementations?

I have wrapped Jackson for JRuby. On github it is called jrjackson.
There is also a ruby gem.

It is about a lot faster.

Davids benchmarks on my machine:

Writing
user system total
real
jrjackson: 0.585000 0.000000 0.585000 ( 0.585000)

Reading
user system total
real
jrjackson: 0.621000 0.000000 0.621000 ( 0.621000)

vs

1.9.3

Writing
user system total
real
oj: 2.910000 0.000000 2.910000 (
2.915350)
yajl: 2.160000 0.000000 2.160000 (
2.166557)
json_gem: 2.190000 0.000000 2.190000 ( 2.199133)
json_pure: 2.200000 0.000000 2.200000 ( 2.204436)

Reading
user system total
real
oj: 1.290000 0.000000 1.290000 (
1.291839)
yajl: 2.130000 0.000000 2.130000 (
2.138048)
json_gem: 2.400000 0.020000 2.420000 ( 2.418907)
json_pure: 2.360000 0.020000 2.380000 ( 2.393626)

This is similar to my own benchmarks which use a much larger hash.

The main problem with jrjackson is that it only works with hashes or
arrays.
I have not found a method to leverage jacksons ability to de/serialise
a ruby object as if it was a java bean or using annotations.

Jrjackson is good for boundary conversions to and from JSON. It also
includes de/serialisation to Smile binary representations.

I developed it to pass hash data around in ZMQ payloads. FWIW I
created a wrapper around the JNI based jzmq library too -
unsurprizingly called jrzmq in github.

Guy

5

On Thu, Apr 19, 2012 at 12:13 PM, Guy B. [email protected]
wrote:

It is about a lot faster.

Strange. I was curious, since I believe I could gain some milliseconds
on my json generation in my sinatra webapp.

So I downloaded jackson-all-1.9.6.jar to compare it with installed gem
json-1.5.3-java. I ripped a json somewhere, made it a hash and tried
this simple code. I did not try jrjackson, as I wanted to compare java
extensions’ performances.

code1: json-1.5.3-java

a = {“name”=>{“first”=>“Joe”, “last”=>“Sixpack”}, “gender”=>“MALE”,
“verified”=>false, “userImage”=>“Rm9vYmFyIQ==”}
require ‘json’
require ‘benchmark’
Benchmark.bm do |x|
x.report { 1000.times { a.to_json } }
end

  user     system      total        real

0.019000 0.000000 0.019000 ( 0.019000)

code2: jackson (maybe my code is not efficient enough? I’m opened to
improvements)

a = {“name”=>{“first”=>“Joe”, “last”=>“Sixpack”}, “gender”=>“MALE”,
“verified”=>false, “userImage”=>“Rm9vYmFyIQ==”}

require ‘java’
java_import org.codehaus.jackson.JsonFactory
java_import org.codehaus.jackson.map.ObjectMapper
class Hash
def to_json
factory = JsonFactory.new
mapper = ObjectMapper.new
writer = java.io.StringWriter.new
generator = factory.createJsonGenerator writer
mapper.writeValue generator, self
generator.close
return writer.toString
end
end
require ‘benchmark’
Benchmark.bm do |x|
x.report { 1000.times { a.to_json } }
end

  user     system      total        real

3.295000 0.000000 3.295000 ( 3.295000)

150x faster on this simple testcase to use json-1.5.3-java than using
jackson. If you’ve code improvements or if I made obvious mistakes,
thanks to let me know.

Feeling puzzled then by the original report and the latest inputs on
jackson…

–
Christian

6

Also… with Jackson creating an JsonFactory/ObjectMapper is an
expensive operation - they should be created once and reused between
iterations.

I also recommend using the method:

String str = mapper.writeValueAsString(obj);

to write the object as a string in one operation

On 19/04/12 11:36, Christian MICHON wrote:

code1: json-1.5.3-java
0.019000 0.000000 0.019000 ( 0.019000)
class Hash
require ‘benchmark’

Feeling puzzled then by the original report and the latest inputs on jackson…

–
Tim F.

Vert.x - effortless polyglot asynchronous application development

twitter:@timfox

7

On Thu, Apr 19, 2012 at 1:06 PM, Tim F. [email protected] wrote:

Also… with Jackson creating an JsonFactory/ObjectMapper is an expensive
operation - they should be created once and reused between iterations.

I also recommend using the method:

String str = mapper.writeValueAsString(obj);

to write the object as a string in one operation

Thanks Tim. I’ll try these…

–
Christian

8

In Java benchmarks I always run them for an extended period of time
(i.e. minutes) to minimise indeterminacies due to gc, and also give time
for the jit to kick in.

I don’t think 1000 iterations is anywhere near enough to get a reliable
figure.

I’m not saying that’s explains the low results, but it’s more of a
general point about conducting micro-benchmarks on the JVM.

On 19/04/12 11:36, Christian MICHON wrote:

code1: json-1.5.3-java
0.019000 0.000000 0.019000 ( 0.019000)
class Hash
require ‘benchmark’

Feeling puzzled then by the original report and the latest inputs on jackson…

–
Tim F.

Vert.x - effortless polyglot asynchronous application development

twitter:@timfox

9

On Thu, Apr 19, 2012 at 1:18 PM, Christian MICHON
[email protected] wrote:

Thanks Tim. I’ll try these…

Good. Now the 2nd snippet looks like this:

a = {“name”=>{“first”=>“Joe”, “last”=>“Sixpack”}, “gender”=>“MALE”,
“verified”=>false, “userImage”=>“Rm9vYmFyIQ==”}
require ‘java’
$mapper = org.codehaus.jackson.map.ObjectMapper.new # ugly I know…
class Hash
def to_json
$mapper.writeValueAsString self
end
end
require ‘benchmark’
Benchmark.bm do |x|
x.report { 1000.times { a.to_json } }
end

I then increased to 10,000,000 for both snippets to have a decent
comparison:

c:\jruby>jruby -b json1.rb (json 1.5.3 java)
user system total real
23.013000 0.000000 23.013000 ( 23.013000)
Runtime: 23433 ms

c:\jruby>jruby -b json2.rb (jackson)
user system total real
65.795000 0.000000 65.795000 ( 65.795000)
Runtime: 66123 ms

json 1.5.3 java is still faster… Seen on jruby 1.6.7
(ruby-1.8.7-p357) (2012-02-22 3e82bc8) (Java HotSpot™ Client VM
1.6.0_31) [Windows 7-x86-java]

–
Christian

10

Hmm I think this forum’s email notification is broken.
Anyhoo, wow thanks for all the information guys!

Charlie’s explanations made a lot of sense, and the JrJackson lib. WOW!
It’s ridiculously fast! This thing is good! I sent a pull request to
multi_json. Wish I’d known about it before I did that benchmarking
blogpost. Glad I know now though :slight_smile:

Looking forward to JRuby 1.7.0 btw :smiley:
What a solid product!