Troubleshooting Spree on jRuby

alcina · September 26, 2013, 1:39am

I’ve been trying for a long time to get Spree
http://spreecommerce.com/running on jRuby and we’ve landed a bunch
of commits to get to the point
where the app will boot. I’ve been running locally on my Mac with Spree
2.0.4/Rails 3.2.14/Puma 2.6 with jRuby 1.7.4 in Ruby 1.9.3 mode locally,
both with config.threadsafe! on & off. All has been well, so I attempted
to
deploy to Heroku, which is where things went off the rails. Many times
the
app times out on boot, which is frustrating, but when it does boot, I
get
the following very bizarre crash:

gist.github.com

https://gist.github.com/swrobel/ea48858e421b5ceb2a43

gistfile1.txt

NoMethodError: super: no superclass method `inherited_without_kaminari'
                                   method_missing at org/jruby/RubyBasicObject.java:1696
                                   method_missing at /app/vendor/bundle/jruby/1.9/gems/activerecord-3.2.14/lib/active_record/dynamic_matchers.rb:55
                                        inherited at /app/vendor/bundle/jruby/1.9/gems/activerecord-3.2.14/lib/active_record/base.rb:398
                          inherited_with_kaminari at /app/vendor/bundle/jruby/1.9/gems/kaminari-0.14.1/lib/kaminari/models/active_record_extension.rb:10
                                            Spree at /app/vendor/bundle/jruby/1.9/gems/spree_core-2.0.4/app/models/spree/role.rb:2
                                           (root) at /app/vendor/bundle/jruby/1.9/gems/spree_core-2.0.4/app/models/spree/role.rb:1
                                          require at org/jruby/RubyKernel.java:1054
                                          require at /app/vendor/bundle/jruby/1.9/gems/polyglot-0.3.3/lib/polyglot.rb:63
                                          require at /app/vendor/bundle/jruby/1.9/gems/activesupport-3.2.14/lib/active_support/dependencies.rb:251

This file has been truncated. show original

I’ve tried w/ config.threadsafe! on & off and it occurs in both cases,
so
it doesn’t seem to be a threadsafety issue. I’ve tried with both
Kaminari
(Spree dependency) versions 0.14.1 (latest gem) and github master which
seems to have some threadsafety fixes. Not quite sure how to triage
since I
don’t seem able to reproduce locally no matter what I try.

I figured this was just pushing too hard to get something to run on
jRuby
that just isn’t meant for it, but listening to Charles Nutter’s Ruby
Rogues
appearance inspired me to open the discussion here to see if anyone can
help triage. I definitely thing jRuby is what Spree needs to run
manageably
at scale.

Stefan_W · September 26, 2013, 2:48am

It definitely sounds like a thread-safety issue to me, which is probably
why you’re having such a hard time replicating it. It is definitely the
Right Thing ™ to use config.threadsafe!, but not entirely surprising
that it doesn’t solve all threading concerns. I’m curious to know if
that
backtrace was from a time when you had config.threadsafe! on or off.

Stefan_W · September 26, 2013, 3:26am

This looks like it is a bug and is already fixed in jruby master and
will be included in jruby 1.7.5:

github.com/jruby/jruby

super inside def self.new behaves incorrect when called via alias

opened 07:40PM - 29 Jul 13 UTC

closed 02:38AM - 21 Aug 13 UTC

mbj

When defining `new` on a class singleton that gets aliased to another name, `sup…er` inside the class dispatches to the aliased, not the original name. MRI and RBX both behave different here and dispatch to superclasses original named method. crash.rb ``` ruby class Foo def self.new super end class << self alias_method :bar, :new end end p Foo.bar ``` jruby: `jruby 1.7.4 (1.9.3p392) 2013-05-16 2390d3b on OpenJDK 64-Bit Server VM 1.7.0_40-b31 +indy [linux-am` ``` mbj@mbj ~/devel/crash (master*) % jruby crash.rb NoMethodError: super: no superclass method `bar' new at crash.rb:4 (root) at crash.rb:13 ``` cruby: `ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]` ``` mbj@mbj ~/devel/crash (master*) % ruby crash.rb #<Foo:0x007f0f1d4fc580> ``` I reduced this bug from around 20kloc, and hope you can fix it in the next minor release. I can trigger this very easily from my production code. Using 1.7.2 now where this bug does not exist.

Stefan_W · September 26, 2013, 3:30am

Also, I think you can work around this bug by disabling invokedynamic:

Stefan_W · September 26, 2013, 5:22am

Ah, JIT issue, not thread issue. Try heroku config:add
JRUBY_OPTS=“-Xcompile.invokedynamic=false” (from the issue Alex linked)
and
see if that fixes your heroku deploy, leaving config.threadsafe! in
place.
Also, I think you can work around this bug by disabling invokedynamic:

On Sep 25, 2013, at 9:25 PM, Alex T. [email protected]
wrote:

This looks like it is a bug and is already fixed in jruby master and
will
be included in jruby 1.7.5:

github.com/jruby/jruby

super inside def self.new behaves incorrect when called via alias

opened 07:40PM - 29 Jul 13 UTC

closed 02:38AM - 21 Aug 13 UTC

mbj

When defining `new` on a class singleton that gets aliased to another name, `sup…er` inside the class dispatches to the aliased, not the original name. MRI and RBX both behave different here and dispatch to superclasses original named method. crash.rb ``` ruby class Foo def self.new super end class << self alias_method :bar, :new end end p Foo.bar ``` jruby: `jruby 1.7.4 (1.9.3p392) 2013-05-16 2390d3b on OpenJDK 64-Bit Server VM 1.7.0_40-b31 +indy [linux-am` ``` mbj@mbj ~/devel/crash (master*) % jruby crash.rb NoMethodError: super: no superclass method `bar' new at crash.rb:4 (root) at crash.rb:13 ``` cruby: `ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]` ``` mbj@mbj ~/devel/crash (master*) % ruby crash.rb #<Foo:0x007f0f1d4fc580> ``` I reduced this bug from around 20kloc, and hope you can fix it in the next minor release. I can trigger this very easily from my production code. Using 1.7.2 now where this bug does not exist.

On Sep 25, 2013, at 8:47 PM, Anthony J. [email protected] wrote:

It definitely sounds like a thread-safety issue to me, which is probably
why you’re having such a hard time replicating it. It is definitely the
Right Thing ™ to use config.threadsafe!, but not entirely surprising
that it doesn’t solve all threading concerns. I’m curious to know if
that
backtrace was from a time when you had config.threadsafe! on or off.

Stefan_W · September 26, 2013, 6:17pm

Thanks for the suggestion! JRUBY_OPTS=“-Xcompile.invokedynamic=false”
did
indeed fix the issue and the app serves requests. However, I was under
the
impression that invokedynamic was disabled by default on jruby
1.7.4/openjdk 7. Is that not the case?

Now on to troubleshooting why the app will only boot within Heroku’s 60s
limit 1 out of 10 times… Any suggestions there?

–
Stefan W.
CTO, CultCosmetics.com https://www.cultcosmetics.com
(949) 394-8728

Stefan_W · September 26, 2013, 6:33pm

On Sep 26, 2013, at 12:07 PM, Stefan W. [email protected]
wrote:

Thanks for the suggestion! JRUBY_OPTS=“-Xcompile.invokedynamic=false” did indeed
fix the issue and the app serves requests. However, I was under the impression
that invokedynamic was disabled by default on jruby 1.7.4/openjdk 7. Is that not
the case?

It was enabled for 7u40 and up on 1.7.4, in the hopes that it was going
to be stable enough. We were wrong. 1.7.5 will disable InvokeDynamic on
all Java 7 versions.

Stefan_W · September 26, 2013, 6:35pm

On Sep 26, 2013, at 12:25 PM, Hirotsugu A. [email protected]
wrote:

On Sep 26, 2013, at 12:07 PM, Stefan W. [email protected] wrote:

Thanks for the suggestion! JRUBY_OPTS=“-Xcompile.invokedynamic=false” did
indeed fix the issue and the app serves requests. However, I was under the
impression that invokedynamic was disabled by default on jruby 1.7.4/openjdk 7. Is
that not the case?

It was enabled for 7u40 and up on 1.7.4, in the hopes that it was going to be
stable enough. We were wrong. 1.7.5 will disable InvokeDynamic on all Java 7
versions.

disable by default