Follow-up JRuby 1.7.0 performance question

This was prompted by reading the Topic headed “jruby 1.7.0 with
invokedynamic.all=true on HotSpot not faster than 1.6.8?”

Am I correct in thinking that the “speedup” parts of the JVM (including
invokeDynamic) only makes a difference for a loop of code that is
repeated so many times that it takes 2 minutes to run?

There can’t be too many cases like that about. I have one program that
simulates charging and discharging lead-acid batteries and the whole
thing only takes about a minute to iterate over 6 batteries at simulated
ten minute intervals for 3 years.

Most of my programs have methods that run once and present a new screen
to the user e.g. downloading and displaying weather data.

Does this mean that my programs would see no significant performance
improvement with 1.7.x compared with 1.6.8?

…R

In broad strokes, if your program has no ‘hot’ code - e.g. methods
that are called many, many times, then a JIT has no effect on your
code.

When benchmarking, we generally try to run loops for a long time (e.g.
100 million times or more), and repeat them, to ensure that the code
has been compiled, and to remove any transient system load effects.

If your code is running hard for 1 minute (and not sleeping or doing
I/O), then there is a good possibility it will benefit from some
speedup. On my ancient macbook, I get in the order of 30 million
function invocations/second, code running for a minute could have 1.8
billion+ method calls - so there’s a good chance a lot of those will
be optimizable.

Invoke dynamic is used for more than method calls, so even things like
constant and global variable reads will be improved with 1.7.x.

TL;DR - try it and see. Theorising on performance never works.
Actual results are what you need.

Thanks Wayne,
I get the impression from your reply that I am unlikely to see much of
an improvement. I’m reminded of the adage “if you have to measure it,
you haven’t made a difference”.

I am not interested to test my existing programs with different versions
of JRuby - they perform well enough.

I am hoping to find the value of X in the statement “1.7.x runs X%
faster than 1.6.8” as one of the factors in a decision about which
version to use.

Before this I had the impression (from various bits I had read) that X
would be a substantial number - but it looks I misunderstood.

…R

Wayne M. wrote in post #1093941:
In broad strokes, if your program has no ‘hot’ code - e.g. methods
that are called many, many times, then a JIT has no effect on your
code.

When benchmarking, we generally try to run loops for a long time (e.g.
100 million times or more), and repeat them, to ensure that the code
has been compiled, and to remove any transient system load effects.

If your code is running hard for 1 minute (and not sleeping or doing
I/O), then there is a good possibility it will benefit from some
speedup. On my ancient macbook, I get in the order of 30 million
function invocations/second, code running for a minute could have 1.8
billion+ method calls - so there’s a good chance a lot of those will
be optimizable.

Invoke dynamic is used for more than method calls, so even things like
constant and global variable reads will be improved with 1.7.x.

TL;DR - try it and see. Theorising on performance never works.
Actual results are what you need.

Robin -

JIT (aka “hot spot”) compiling is just one of several things that can
affect performance factors in JRuby applications.

On other message threads in this forum, we’ve seen how changing a single
line of code can potentially have a tremendous effect on program
throughput. More below…

On Jan 27, 2013, at 6:52 AM, Robin McKay [email protected] wrote:

Thanks Wayne,
I get the impression from your reply that I am unlikely to see much of
an improvement. I’m reminded of the adage “if you have to measure it,
you haven’t made a difference”.

There are lots of variables in play, many known only by those who are
expert in JRuby and JVM internals. And those variables have different
degrees of effect depending on the runtime behavior of a program. That’s
why real world testing is important.

I am not interested to test my existing programs with different versions
of JRuby - they perform well enough.

If you’re happy with your current performance, then fine, but do realize
that by turning away from the issue you may be missing opportunities for
improvement, possibly dramatic improvements.

I am hoping to find the value of X in the statement “1.7.x runs X%
faster than 1.6.8” as one of the factors in a decision about which
version to use.

While it’s our job to analyze, detect patterns, and make informed
judgments based on them, we need to be careful not to oversimplify.
While there may be a single value for X as a very coarse indicator, the
real value you encounter may vary wildly depending on the runtime
behavior of your program.

It seems that you are looking for shortcuts and generalizations.
There’s nothing wrong with that, it’s our job to accomplish as much as
possible in as little time as possible. However, when it comes to
JRuby/Java performance, shortcuts and generalizations are often
unreliable.

  • Keith

Keith R. Bennett

Thanks Keith,

I understand perfectly what you say about performance varying widely
between cases. Nevertheless a generalized indication (perhaps a range)
would be very useful when each new version comes out (especially if
there are “dramatic improvements” to be had). You might be able to say
something like this:
“this changes in this version don’t effect performance”
or
“in most cases this version should give a 5% performance improvement but
special attention has been given to XX and if you use that feature you
should see a 50% speed improvement in that part of your code”

Obviously another way of looking at things is if your program is
sluggish try it on a newer version to see if that improves things.

Interestingly I don’t think I have ever seen advice about what things to
do or not to do in my JRuby code to minimize runtime.

And, of course, there may be many non-speed reasons to justify an
upgrade.
…R

Keith B. wrote in post #1093973:
Robin -

If you’re happy with your current performance, then fine, but do realize
that by turning away from the issue you may be missing opportunities for
improvement, possibly dramatic improvements.

While it’s our job to analyze, detect patterns, and make informed
judgments based on them, we need to be careful not to oversimplify.
While there may be a single value for X as a very coarse indicator, the
real value you encounter may vary wildly depending on the runtime
behavior of your program.

It seems that you are looking for shortcuts and generalizations.
There’s nothing wrong with that, it’s our job to accomplish as much as
possible in as little time as possible. However, when it comes to
JRuby/Java performance, shortcuts and generalizations are often
unreliable.

  • Keith

Keith R. Bennett
Keith R. Bennett - Reston, Virginia, Bennett Business Solutions Inc | about.me

It sounds like, for you, the main advice for upgrading will be that
there will never be a JRuby 1.6.9. If you run into a problem with
1.6.8 you will either have to upgrade or patch it yourself. If your
code never changes then perhaps even this is not a factor, but most
people do change their code and run into the unexpected. I typically
try and keep up so I am upgrading on my schedule (and terms) instead
of immediate need (e.g. hit a bug).

An interesting realization is that language runtimes are less prone to
upgrade rot than other programs because the behavior is so well
defined. If you wait two major versions, then it may be zero pain to
upgrade. This is rarely true with most software…

-Tom

On Mon, Jan 28, 2013 at 2:30 AM, Robin McKay [email protected]
wrote:

special attention has been given to XX and if you use that feature you
…R

While there may be a single value for X as a very coarse indicator, the

http://xircles.codehaus.org/manage_email


blog: http://blog.enebo.com twitter: tom_enebo
mail: [email protected]

Great discussion!

Im going to try to bring my contribution…

Im implementing a multi-dimensional array class (MDArray), which by the
way, Im thinking to give to the community if there is any interest (more
on another thread) in line with Numpy. The solution uses
multi-dimensional
java array implemented by unidata Java NetCDF library. This library is
freely available and the source code is released under the (MIT-style)
netCDF C library
licensehttp://www.unidata.ucar.edu/software/netcdf/copyright.html.
Ive being doing some performance testing and would like to share the
results. My machine: Intel Core i5-2400 CPU @ 3.10GHz. 4,00 GB Windows
7
64 on top of cygwin.

The code bellow creates a new multi-dimensional array of type double
with
4 dimensions with the sizes of 7, 500, 20, 320. So, the total number of
elements in the array is: 7 x 500 x 20 x 320 = 22.400.000. The
fromfunction block receives the dimensions and fills the given element
with
the resulting value. So in the example bellow @a[5, 10, 15, 20] = 5 +
10 +
15 + 20 = 50.

@a = MDArray.fromfunction(“double”, [7, 500, 20, 320]) do |x, y, z, k|

    x + y + z + k

end

The relevant ruby code called to execute this method is:

def set_block(*args)

get_args(*args) do |op_iterator, shape, *other_args|

  block = other_args[0]

  while (op_iterator.has_next?)

    op_iterator.next

    op_iterator.set_current(block.call(op_iterator.get_current_counter))

  end if block

end

end

     Get_args just parses the arguments to fromfunction.  In this 

case
the argument is the multi-dimension array @a and it calls the internal
block giving an iterator over @a op_iterator, the shape of the array
and
the remaining args if there are any;

     We then iterate over all elements of @a and set the current 

value
by calling the block with the current_counter value, e.g., [0, 0, 0, 0],
[0, 0, 0, 1], etc.

The methods set_current and get_current_counter are calls to the NetCDF
java methods and should be fairly fast.

Running this code with Jruby 1.6.8 as: /jruby-1.6.8/bin/jruby --server
-J-Djruby.compile.frameless=true -J-Djruby.compile.fastops=true
-J-Xmn512m
-J-Xms1024m -J-Xmx1024m $1 takes between 36 and 38 seconds.

Running the same code with Jruby 1.7.2 as: /jruby-1.7.2/bin/jruby
–server
-Xinvokedynamic.constants=true -J-Xmn512m -J-Xms1024m -J-Xmx1024m $1
takes
between 33 and 35 seconds
.

So, there is an improvement, but I wouldnt say it is very large and I
dont know if we can make any generalization.

35 seconds is actually a lot of time and in order to improve on this I
created java methods to execute the loop.

This is the Java method that does the same loop as the ruby method
above:

public static void setAll4(ArrayDouble array, D4 func) {

            IndexIterator iterator = array.getIndexIterator();

            int[] counter;

            while (iterator.hasNext()) {

                iterator.next();

                counter = iterator.getCurrentCounter();

                iterator.setDoubleCurrent(func.call(counter[0],

counter[1], counter[2],

counter[3]));

            }

}

Unfortunately I had to create methods called setAll1, setAll2, setAll3
setAll7 for efficiency reasons as I dont think there is a way for Jruby
to
finding the proper method. In the example above, setAll4 will be called
as
my array is 4 dimensions.

Now running this code with Jruby 1.6.8 with the same flags as before
executes in 4.23 seconds with very minor differences between runs. So,
bringing the code to Java does actually make a huge difference.

Now, with Jruby 1.7.2 with invokedynamics it executes in 4.8 to 5.1
seconds. So, actually it performs worst than 1.6.8. Even changing the
flags to the same flags as in 1.6.8 does not improve performance.

So any comments and ideas why 1.7.2 is worst than 1.6.8 when the loop is
in
Java?

Are there other interesting flags that should be used in order to
improve
performance?

Thanks for all the comments and ideas…

Rodrigo

To be a little more on topic, performance in 1.7.x is a moving target.
If we could claim a percentage it would make things simpler, but
unfortunately it is not that simple.

There are new improvements per point release like Charlie making
constant access nearly free for 1.7.2.

There is also changing performance numbers per ‘u’ release of the JVM
since the JVM engineers are actively working on invokedynamic (+ other
areas). Invokedynamic gets special mention since they changes have
been substantial and it is still considered a new feature of the JVM.
Indy keeps improving but it is definitely changing each VM update.

The general conclusion to all ‘is it worth it’ threads is: try it and
see. I know it takes work to try, but it is impossible to distill
this. I have heard people talk about 4-6x improvement to people
saying things have gotten 10-15% slower than 1.6.8.

-Tom

On Mon, Jan 28, 2013 at 11:42 AM, Robin McKay [email protected]
wrote:

is no need to upgrade a working application. But I might take the
been achieved.

try and keep up so I am upgrading on my schedule (and terms) instead
wrote:


blog: http://blog.enebo.com twitter: tom_enebo
mail: [email protected]

Thanks Tom,

I think you are off-Topic as you have not commented on speed
differences:) I am well aware there are other issues such as elimination
of bugs and security issues that justify upgrades. But my question was
about speed - and I guess you know more about that than anyone else.

I have been using 1.7.x since it appeared, but I am now wondering if I
really needed to change. I have myself organized so each project has its
own copy of JRuby so versions and gems are completely independent. There
is no need to upgrade a working application. But I might take the
trouble to do so if there was a 30% speed increase!

This has really been prompted because I can get 1.6.8 to work on an
Android device, but not 1.7.x and, in any case, 1.6.8 is considerably
smaller.

Generally when a new version comes out the list of changes is pretty
useless for deciding if the upgrade is valuable unless you have
experienced one of the problems that it addresses and, as far as I know,
there is never any advice about the performance improvement that has
been achieved.

…R

Thomas E Enebo wrote in post #1094092:
It sounds like, for you, the main advice for upgrading will be that
there will never be a JRuby 1.6.9. If you run into a problem with
1.6.8 you will either have to upgrade or patch it yourself. If your
code never changes then perhaps even this is not a factor, but most
people do change their code and run into the unexpected. I typically
try and keep up so I am upgrading on my schedule (and terms) instead
of immediate need (e.g. hit a bug).

An interesting realization is that language runtimes are less prone to
upgrade rot than other programs because the behavior is so well
defined. If you wait two major versions, then it may be zero pain to
upgrade. This is rarely true with most software…

-Tom

On Mon, Jan 28, 2013 at 2:30 AM, Robin McKay [email protected]
wrote:

special attention has been given to XX and if you use that feature you
…R

While there may be a single value for X as a very coarse indicator, the

http://xircles.codehaus.org/manage_email


blog: http://blog.enebo.com twitter: tom_enebo
mail: [email protected]