I’m having a really strange issue with Tomcat 6(newest release, although
I’ve tried several releases from the 6 and 7 branches). I have two JRuby
on Rails apps running on a linux(CentOS) server. Every couple of days,
all Tomcat apps become unreachable. The Java process shoots to the top
of my running process list.
I have no idea which app is causing this problem, but it takes down
Tomcat completely. (I thought each app was running in its own space).
The worst part is NOTHING out of the ordinary is reported in the logs.
I’m guessing one of the apps is leaking memory, but I have no idea where
to even start looking. I’m not a java guy in the slightest, so I’m not
familiar with the ecosystem and how to look into this.
I’ve tried multiple JRuby versions/multiple Rails versions/multiple
Tomcat versions, even multiple Java versions (all 1.6.x Sun Hotspot).
How do I go about troubleshooting this?
Thanks,
Jim W.
Hi Jim,
What’s your permgen/max heap set to?
256/768 seems to work nicely.
Also, have you isolated the apps to make sure there isn’t a memory leak?
Lastly, I assume you’re building these via warble to run in Tomcat?
Do you have your min and max runtimes fixed? Otherwise the pool will
grow out of control.
Jeffrey L. wrote in post #1034155:
Hi Jim,
What’s your permgen/max heap set to?
256/768 seems to work nicely.
Also, have you isolated the apps to make sure there isn’t a memory leak?
Lastly, I assume you’re building these via warble to run in Tomcat?
Do you have your min and max runtimes fixed? Otherwise the pool will
grow out of control.
Most of this is above my head in the Java world. (I have no problem with
the actual language… it’s all the configuration of other java related
stuff JVM settings, etc that is beyond me.)
My JVM runtime settings are:
JAVACMD_OPTS="-server -Xmx512M
-XX:+UseConcMarkSweepGC-XX:OnOutOfMemoryError=“mail -s 'OOM on
hostname at date’ [email protected] <<< ‘’”"
Where can I set min/max runtimes? I set the above settings in
/etc/java/java.conf. I can’t even be sure it’s getting applied. Still
poking around.
Yes, I’m building these via Warbler.
Is there a way to isolate these apps without simply shutting one down? I
can ask google if it’s not a simple answer.
After playing with some JAVA_OPTS in my Tomcat init script, I am getting
this in the catalina.out:
SEVERE: The web application [/isr] created a ThreadLocal with key of
type [org.jruby.RubyEncoding$2] (value
[org.jruby.RubyEncoding$2@1e185b9]) and a value of type
[org.jruby.RubyEncoding.UTF8Coder] (value
[org.jruby.RubyEncoding$UTF8Coder@6aa50f]) but failed to remove it when
the web application was stopped. This is very likely to create a memory
leak.
At least that’s something to start working with.
-Jim
Btw it’s well worth installing visualvm to debug stuff like this.
It’s fairly easy to setup, and then you can look at thread dumps and
see where the problem is.
Just the other day we started having threads just randomly start
chewing up cpu and being in a constant run state. A quick thread dump
showed we had a bad memcache key that was causing contention. Would
have taken hours to track that down without a thread dump.
Chris
You can set min/Max runtimes in warble.rb config file in your app config
directory (jruby.min.runtime…). I usually set it to min=2, max=4.
We had a similar issue some time back when using JRuby on Rails. The
best JRuby & Rails configuration that worked for us from reliability and
scalability point of view is:
- Set JRuby min and max runtimes to 1
- Enable config.threadsafe! options in Rails
- This enables the application to make the most of JVM real threads
without using too much memory. You are now running a single Ruby/Rails
runtime/process to serve all incoming requests.
- Make sure your code is thread safe.
- Use jvisualvm to monitor the app & do root cause analysis in case
problem persists.
Hope this helps.