Hi,
We are running a Rails app on Torquebox, and we see a intermittentissue
where some of the servers which are not under load show a high
CPU(100-800% - 16 CPU server).
I dumped the threads, and the active threads all seem to contain a
similar trace. In fact the number of these threads isproportionalto the
percent of CPU consumed. For example some servers are a 100% CPU and
there is one of these threads, some are at 600% with 6 threads.
We areusingJRuby 1.7.3 with the jruby.compile.mode=JIT. Java version
is1.7.0_10
Here are the two different types of threads I see:
As you can see we are using NewRelic.
Thread 3433: (state = IN_JAVA)
- java.lang.String.valueOf(int) @bci=1, line=2959 (Compiled frame;
information may be imprecise)
com.newrelic.agent.instrumentation.ClassloaderJarExtractor.isNewLoader(java.lang.ClassLoader)
@bci=19, line=102 (Compiled frame)
com.newrelic.agent.instrumentation.ClassloaderJarExtractor.grabJarInformation(java.lang.ClassLoader)
@bci=21, line=56 (Interpreted frame)
com.newrelic.agent.instrumentation.ClassTransformer.transform(java.lang.ClassLoader,
java.lang.String, java.lang.Class, java.security.ProtectionDomain,
byte[]) @bci=5, line=150 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader,
java.lang.String, java.lang.Class, java.security.ProtectionDomain,
byte[]) @bci=50, line=188 (Compiled frame) - sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader,
java.lang.String, java.lang.Class, java.security.ProtectionDomain,
byte[], boolean) @bci=34, line=424 (Compiled frame) - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int,
java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted
frame) - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int,
java.security.ProtectionDomain) @bci=30, line=791 (Interpreted frame)
org.jruby.util.ClassCache$OneShotClassLoader.defineClass(java.lang.String,
byte[]) @bci=9, line=73 (Interpreted frame)
org.jruby.util.ClassCache.defineClass(org.jruby.util.ClassCache$ClassGenerator)
@bci=108, line=128 (Interpreted frame)
- org.jruby.util.ClassCache.cacheClassByKey(java.lang.Object,
org.jruby.util.ClassCache$ClassGenerator) @bci=57, line=96 (Interpreted
frame) - org.jruby.compiler.JITCompiler$1.run() @bci=304, line=167 (Interpreted
frame) - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471
(Compiled frame) - java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334
(Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted
frame)
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=46, line=1110 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603
(Interpreted frame) - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
Thread 3434: (state = IN_JAVA)
- java.util.HashMap.getEntry(java.lang.Object) @bci=72, line=446
(Compiled frame; information may be imprecise) - java.util.HashMap.get(java.lang.Object) @bci=11, line=405 (Compiled
frame)
com.newrelic.agent.instrumentation.ClassloaderJarExtractor.isNewLoader(java.lang.ClassLoader)
@bci=28, line=103 (Compiled frame)
com.newrelic.agent.instrumentation.ClassloaderJarExtractor.grabJarInformation(java.lang.ClassLoader)
@bci=21, line=56 (Interpreted frame)
com.newrelic.agent.instrumentation.ClassTransformer.transform(java.lang.ClassLoader,
java.lang.String, java.lang.Class, java.security.ProtectionDomain,
byte[]) @bci=5, line=150 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader,
java.lang.String, java.lang.Class, java.security.ProtectionDomain,
byte[]) @bci=50, line=188 (Compiled frame) - sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader,
java.lang.String, java.lang.Class, java.security.ProtectionDomain,
byte[], boolean) @bci=34, line=424 (Compiled frame) - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int,
java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted
frame) - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int,
java.security.ProtectionDomain) @bci=30, line=791 (Interpreted frame)
org.jruby.util.ClassCache$OneShotClassLoader.defineClass(java.lang.String,
byte[]) @bci=9, line=73 (Interpreted frame)
org.jruby.util.ClassCache.defineClass(org.jruby.util.ClassCache$ClassGenerator)
@bci=108, line=128 (Interpreted frame)
- org.jruby.util.ClassCache.cacheClassByKey(java.lang.Object,
org.jruby.util.ClassCache$ClassGenerator) @bci=57, line=96 (Interpreted
frame) - org.jruby.compiler.JITCompiler$1.run() @bci=304, line=167 (Interpreted
frame) - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471
(Compiled frame) - java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334
(Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted
frame)
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=46, line=1110 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603
(Interpreted frame) - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
Has any one seen this, or have any thoughts?
In the past we have seen similar issues running an app with
thejruby.compile.mode set toJIT, changing it to FORCE seemed to avoid(or
move) the issue, but we are hoping to use JIT. How safe is the JIT mode
in a multithreaded environment?
Thanks,
Tom