More granular Benchmark times

Ruby JRuby
1

Here’s a thought: would it be possible to use the equivalent of
System.nano_time within the Benchmark module, thus making it more
accurate?
That might be nicer by default.

2

On Jul 12, 2010, at 4:39 PM, Roger P. wrote:

Here’s a thought: would it be possible to use the equivalent of
System.nano_time within the Benchmark module, thus making it more
accurate?
That might be nicer by default.

Unless things are changing in 1.6 and later, JRuby has time granularity
measured in milliseconds. Check for yourself with Time.now.to_f. No
microseconds (let alone nanoseconds) are to be found.

cr

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
3

On Jul 12, 2010, at 7:02 PM, Chuck R. wrote:

On Jul 12, 2010, at 4:39 PM, Roger P. wrote:

Here’s a thought: would it be possible to use the equivalent of
System.nano_time within the Benchmark module, thus making it more
accurate?
That might be nicer by default.

Unless things are changing in 1.6 and later, JRuby has time granularity measured in milliseconds. Check for yourself with Time.now.to_f. No microseconds (let alone nanoseconds) are to be found.

cr

It’s one thing to pursue precision, but how good really is such
benchmarks? In the grand scheme of things, nanoseconds will be swallowed
up by the statistical margin of errors anyway.

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
4

It’s one thing to pursue precision, but how good really is such
benchmarks? In the grand scheme of things, nanoseconds will be swallowed
up by the statistical margin of errors anyway.

In my case it would be to get slightly more accurate measurements of
time blocks, like

total_time = x
total_time += Benchmark.realtime { some more stuff that it would be nice
to measure more precisely }

And why not, really? What’s the use of

Benchmark.realtime { something }
=> 0.0

For me I would prefer more precision

Benchmark.realtime { something }
=> 0.0004235

-r

5

Could you do something like:

loops = 10000 # or whatever
time = Benchmark.realtime (0…loops).each {|m, i| your stuff } / loops

That might help in cancelling out the lack of resolution, but probably
won’t work if you’re not measuring things in isolation.

I’ve been doing benchmarks on jruby something like the gist below, in
conjunction with starting up jruby with --server +X+C (force compilation
and jvm server mode). I also run each bit of code i’m testing enough
times to get a significant time, and then I repeat this around 20 times
to make sure the jvm has warmed up sufficiently. I use this when
optimising, rather than trying to identify bottlenecks.

On Tue, 2010-07-13 at 15:36 +0200, Roger P. wrote:

And why not, really? What’s the use of

Benchmark.realtime { something }
=> 0.0

For me I would prefer more precision

Benchmark.realtime { something }
=> 0.0004235

-r

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
6

On 7/12/10, Roger P. [email protected] wrote:

Here’s a thought: would it be possible to use the equivalent of
System.nano_time within the Benchmark module, thus making it more
accurate?
That might be nicer by default.

Yeah, I’ve thought about this too, but I tend to agree with other
posters who argued that anything less than ms granularity is basically
noise.

You can use FFI to get actual process times, however, which may have
finer granularity:

~/projects/jruby âž” jruby -rffi/times -rbenchmark -e “puts
Benchmark.measure { sleep 1 }”
0.080000 0.000000 0.080000 ( 1.012000)

~/projects/jruby âž” jruby -rbenchmark -e “puts Benchmark.measure { sleep
1 }”
1.057000 0.000000 1.057000 ( 1.010000)

‘ffi/times’ basically uses the native “times” function.

  • Charlie

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
7

Hmm. i would be happy with ms granularity. Perhaps I am missing
something here?

Benchmark.realtime { sleep 0.001}
=> 0.0160000324249268

Appears that one also cannot sleep for < 0.015s in windows:

sleep 0.0001; Benchmark.realtime { 2.times { sleep 0.001 }}
=> 0.0309998989105225

I assume this is also a bug there?
Thanks.
-r

8

I think that is a limitation on java on windows. I seem to recall that
doing System.currentTimeMillis on windows always changed in 16 milli
intervals

On Mon, 2010-07-19 at 14:48 +0200, Roger P. wrote:

=> 0.0309998989105225

I assume this is also a bug there?
Thanks.
-r

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
9

Charles Nutter wrote:

On 7/12/10, Roger P. [email protected] wrote:

Here’s a thought: would it be possible to use the equivalent of
System.nano_time within the Benchmark module, thus making it more
accurate?
That might be nicer by default.

Yeah, I’ve thought about this too, but I tend to agree with other
posters who argued that anything less than ms granularity is basically
noise.

Hmm. i would be happy with ms granularity. Perhaps I am missing
something here?

Benchmark.realtime { sleep 0.001}
=> 0.0160000324249268

C:\dev>c:\dev\ruby\downloads\jruby\bin\jruby -v
jruby 1.6.0.dev (ruby 1.8.7 patchlevel 249) (2010-07-14 fba12c0) (Java
HotSpot™ Client VM 1.6.0_10) [x86-java]

Is this a bug?
-r

10

On Jul 19, 2010, at 8:48 AM, Roger P. wrote:

Nick G. wrote:

I think that is a limitation on java on windows. I seem to recall that
doing System.currentTimeMillis on windows always changed in 16 milli
intervals

Which is why i would suggest they uses nanoTime within the Benchmark
class (and truncate it to ms).

I wanted Time to have more granularity too. Check out this thread from a
year ago to see why it hasn’t happened.

If you come up with a workable patch that resolves the issues with
sleeping/suspending the VM, please submit it.

http://www.ruby-forum.com/topic/188350

cr

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
11

Nick G. wrote:

I think that is a limitation on java on windows. I seem to recall that
doing System.currentTimeMillis on windows always changed in 16 milli
intervals

Which is why i would suggest they uses nanoTime within the Benchmark
class (and truncate it to ms).

-r

12

I wanted Time to have more granularity too. Check out this thread from a
year ago to see why it hasn’t happened.

If you come up with a workable patch that resolves the issues with
sleeping/suspending the VM, please submit it.

A couple of clues appear to be:
http://www.sagui.org/~gustavo/blog/code
http://www.ruby-forum.com/topic/213400#new

so it’s either use LockSupport or call into native code.
Anything potentially acceptable in there?
-r

13

Perhaps LockSupport.parkNanos can work for sleep but it won’t change
the granularity of Date/Time. Still this seems like this will improve
one feature. Anyone want to whip up a patch?

-Tom

On Mon, Jul 19, 2010 at 2:41 PM, Roger P. [email protected]
wrote:

so it’s either use LockSupport or call into native code.

–
blog: http://blog.enebo.com twitter: tom_enebo
mail: [email protected]

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
14

so it’s either use LockSupport or call into native code.
Anything potentially acceptable in there?

Seems to me like using nanoSeconds “if on platform windows” would be all
right, as well.
Also note that (interestingly) the problem seems to only show on XP, not
on Windows 7.
The previous links also have interesting ideas, like using
NtSetTimerResolution.
Cheers!
-rp

15

Thomas E Enebo wrote:

Perhaps LockSupport.parkNanos can work for sleep but it won’t change
the granularity of Date/Time. Still this seems like this will improve
one feature. Anyone want to whip up a patch?

I’ll file a feature request for it on JIRA and hopefully get around to
it.

-r