JRubyClassLoader not getting released

Ruby JRuby
1

We’re running a small JRuby on Rails app under Tomcat 6.0.28 with a
Spring-based backend. I’ve spent some time with the Eclipse Memory
Analysis tool and I can definitely tell that instances of
theJRubyClassLoader are leaking. The total number of classes loaded
(and therefore PermGen usage) goes up every time I deploy. More
importantly, we’re seeing this on our production sites with virtually
every incoming request. (See

for my original question on StackOverflow).

I setup our webapp to on my localhost to only use a single JRuby
runtime and then I effectively did a hot-deploy to Tomcat by touching
the war. After doing that a few times, I can see several instances of
the JRubyClassLoader sitting around.

Since the classloader isn’t getting released, the classes it loaded
aren’t getting released and we’re running out of PermGen space.

Using Eclipse Memory Analysis, I can see the path to the GC root looks
like:

org.jruby.util.JRubyClassLoader
jrubyClassLoader org.jruby.Ruby
runtime org.jruby.util.io.ChannelStream
reference java.lang.ref.Finalizer
next java.lang.ref.Finalizer

And the list of next java.lang.ref.Finalizer goes on seemingly
forever… to point where I can’t seem to find the actual GC root.

If run the Leak Suspects report, the #1 suspect is
“java.lang.ref.Finalizer”, loaded by “”.

Any ideas why the Finalizer is sticking around?

As a possibly related side note, every time I do a hot deploy, a get a
slew of NullPointerExceptions:

java.lang.NullPointerException
at com.kenai.jffi.Function.finalize(Function.java:177)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

And I’m also seeing some errors with ThreadLocals which I suspect are
also related:

SEVERE: A web application created a ThreadLocal with key of type
[null] (value [com.kenai.jaffl.provider.StringIO$1@799a7266]) and a
value of type [java.lang.ref.SoftReference] (value
[java.lang.ref.SoftReference@accc4e9]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[null] (value [org.joni.StackMachine$1@993a289]) and a value of type
[java.lang.ref.WeakReference] (value
[java.lang.ref.WeakReference@3b93d548]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[null] (value [com.kenai.jaffl.provider.StringIO$1@799a7266]) and a
value of type [java.lang.ref.SoftReference] (value
[java.lang.ref.SoftReference@66978886]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[null] (value [org.joni.StackMachine$1@993a289]) and a value of type
[java.lang.ref.WeakReference] (value
[java.lang.ref.WeakReference@66844a08]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[org.springframework.core.NamedThreadLocal] (value [Name of currently
proxied bean]) and a value of type [java.lang.String] (value
[httpConnectionManagerParams]) but failed to remove it when the web
application was stopped. To prevent a memory leak, the ThreadLocal has
been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[null] (value [com.kenai.jaffl.provider.StringIO$1@799a7266]) and a
value of type [java.lang.ref.SoftReference] (value
[java.lang.ref.SoftReference@771887cc]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[null] (value [org.joni.StackMachine$1@993a289]) and a value of type
[java.lang.ref.WeakReference] (value
[java.lang.ref.WeakReference@6b7b9f29]) but failed to remove it when
the web application was stopped. To prevent a memory leak, the
ThreadLocal has been forcibly removed.
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap

All of the initial research was done using JRuby 1.5.0, but I’m seeing
the same results with 1.5.1 despite my hope that the resolution to
JRUBY-4838 would have solved my problems.

Any thoughts or suggestions?

-Sean

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
2

Tomcat does have some know issues that cause memory leaks on hot
deploys due to the WebAppClassLoader not getting garbage collected.
We’ve been running Tomcat for years and generally don’t do hot deploys
on our production environment for this very reason. I’m not convinced
that any given library is necessarily doing anything wrong even if it
triggers the problem. If your app is lean enough that it wouldn’t
otherwise have a problem if not for JRuby then you may be able to get
around it by making sure Jruby is loaded by the shared class loader
instead of the web app class loader.

http://opensource.atlassian.com/confluence/spring/pages/viewpage.action?pageId=2669

From: http://tomcat.apache.org/tomcat-6.0-doc/RELEASE-NOTES.txt

KNOWN ISSUES IN THIS RELEASE:

…

  • Web application reloading and static fields in shared libraries
    -lenny

On Jul 20, 2010, at 9:41 AM, Sean L. wrote:

I setup our webapp to on my localhost to only use a single JRuby
org.jruby.util.JRubyClassLoader

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
[null] (value [com.kenai.jaffl.provider.StringIO$1@799a7266]) and a
the web application was stopped. To prevent a memory leak, the
clearThreadLocalMap
[java.lang.ref.WeakReference] (value
Any thoughts or suggestions?

-Sean

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
3

We generally don’t do hot-deploys either. Hot-deploy is really just
highlighting our underlying problem: JRuby is creating multiple
classloaders
that are not getting unloaded, regardless of hot-deploys.

As I mentioned in the original post, our webapp is JRuby on Rails with a
Spring backend. On our production site, every request that comes in
causes
our loaded class count to increase, which in turn causes PermGen to
increase
and eventually we run out of PermGen. It’s not a matter of just
increasing
PermGen space. On our production server, where we’re not doing
hot-deploys,
I’ve currently got MaxPermGen set at 2GB and over the past 3 days I’ve
seen
a roughly linear increase in loaded classes from 15,000 to 250,000.
PermGen
has gone linearly from 128MB to 1.5GB.

I spent some quality time with JCrawler against the site and produced an
initial memory dump which I analyzed using the Memory Analysis Tool for
Eclipse. That lead me to the results I outlined in my original email. I
then
discovered it was incredibly faster just to do a hot-deploy, which
produces
the exact same problem.

In both cases, classloaders are not getting released. Currently it’s
look
like a Finalizer related to JRuby. Not sure where I can go from there.
Nor
am I even certain I’m on the right path. :frowning: Hence why I posted here.

-Sean

4

I upgraded to JRuby 1.5.1, disabled native/FFI and set the min/max
runtimes
to 1. Unfortunately, the problem remains. :frowning:

I can still see multiple JRubyClassLoaders that didn’t get collected.
The
path to the GC root is the same: jrubyClassLoader ->
org.jruby.util.io.ChannelStream -> java.lang.ref.Finalizer.

Very puzzling.

Any other suggestions?

-Sean

5

Some ideas to try:

  • Turn of native/FFI stuff. I noticed the FFI finalizer in your stack
    trace above. Set -Djruby.native.enabled=false somewhere in Tomcat
    launcher/config files.
  • Make sure you set minimum and maximum # of runtimes with
    Warbler/JRuby-Rack. In Warbler, they’ll be
    config.jruby.(min|max).runtimes in config/warble.rb; inside your war
    file, they will be set inside web.xml.

/Nick

On Tue, Jul 20, 2010 at 11:18 AM, Sean L. [email protected]
wrote:

has gone linearly from 128MB to 1.5GB.
On Tue, Jul 20, 2010 at 11:04 AM, Lenny M. [email protected] wrote:

memory leaks - Track down PermGen problem with JRuby on Rails in Tomcat - Stack Overflow
Using Eclipse Memory Analysis, I can see the path to the GC root looks

at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
[java.lang.ref.SoftReference@accc4e9]) but failed to remove it when
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
[null] (value [org.joni.StackMachine$1@993a289]) and a value of type
application was stopped. To prevent a memory leak, the ThreadLocal has
clearThreadLocalMap
the same results with 1.5.1 despite my hope that the resolution to

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
6

We’ve struggled with webapp leaks in the past. It’s hard to say if
this is a new one or an old one, unfortunately :(. If you can make a
small app you can share that exhibits the behavior and file a bug,
that would help a lot. If you can’t do that but would be willing to
pay for support through Engine Y., we could work with you privately
to help diagnose and fix the issue in your application.

The only other suggestion I have is to try with other containers to
see if they exhibit the memory leak. That would be a good data point
to have actually.

Cheers,
/Nick

On Thu, Jul 22, 2010 at 9:59 AM, Sean L. [email protected]
wrote:

/Nick

increase
initial memory dump which I analyzed using the Memory Analysis Tool for
-Sean

problem. If your app is lean enough that it wouldn’t otherwise have a
From: http://tomcat.apache.org/tomcat-6.0-doc/RELEASE-NOTES.txt

Spring-based backend. I’ve spent some time with the Eclipse Memory
I setup our webapp to on my localhost to only use a single JRuby
org.jruby.util.JRubyClassLoader

at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
[null] (value [com.kenai.jaffl.provider.StringIO$1@799a7266]) and a
the web application was stopped. To prevent a memory leak, the
clearThreadLocalMap
[java.lang.ref.WeakReference] (value
Any thoughts or suggestions?

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
7

odd … we are seeing an ever growing permgen when running under
tomcat that ultimately crashes tomcat as well.

We too are doing some java(ish) stuff within jruby. We’ve been able
to isolate the what is causing our problem…
it is an anonymous class.

our sinatra app uses some JAXB marshaling and we are essentially
calling sort on a Java ArrayList and passing it a block. this block
is an anonymous class and everytime we hit the sort call the PermGen
space has a class added to it. and it’s never released.

I isolated out this test case to reproduce our problem:

run this file like so: jruby -J-XX:+TraceClassLoading
-J-XX:+TraceClassUnloading loader.rb

you can also add the above tracing steps to your specific tomcat
instance (via CATALINA options) to see what is loading continuously
(I would be interested to see if you are getting anything)

require ‘java’
require ‘benchmark’

x = Java::JavaUtil::ArrayList.new

x.add 4
x.add 2
x.add 5

x.each do |item|
p item
end
y = nil
Benchmark.bm do |b|
b.report do
1000000.times do
y = x.sort { |a,b| a <=> b }
end
end
end

y.each do |item|
p item
end

On Thu, Jul 22, 2010 at 1:12 PM, Nick S. [email protected]
wrote:

Any other suggestions?

Warbler/JRuby-Rack. In Warbler, they’ll be

As I mentioned in the original post, our webapp is JRuby on Rails with a
a roughly linear increase in loaded classes from 15,000 to 250,000.
look

running Tomcat for years and generally don’t do hot deploys on our

forever… to point where I can’t seem to find the actual GC root.
at com.kenai.jffi.Function.finalize(Function.java:177)
value of type [java.lang.ref.SoftReference] (value
ThreadLocal has been forcibly removed.
SEVERE: A web application created a ThreadLocal with key of type
[httpConnectionManagerParams]) but failed to remove it when the web
Jul 20, 2010 8:17:43 AM org.apache.catalina.loader.WebappClassLoader
All of the initial research was done using JRuby 1.5.0, but I’m seeing
http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email
8

I originally posted this question on StackOverflow. Today someone posted
a
suggestion that I haven’t tried, but might help you:

Short version is: call to_a on the ArrayList before calling sort.

-Sean

9

FYI, this was reported in http://jira.codehaus.org/browse/JRUBY-4981
and fixed for 1.5.2 and 1.6. We’ll look at pushing out 1.5.2 in the
next week or so.

On Sat, Jul 31, 2010 at 1:23 AM, Sean L. [email protected]
wrote:

wrote:

is an anonymous class and everytime we hit the sort call the PermGen

 p item

small app you can share that exhibits the behavior and file a bug,

Any other suggestions?

  • Make sure you set minimum and maximum # of runtimes with

classloaders
hot-deploys,
Eclipse. That lead me to the results I outlined in my original
am I even certain I’m on the right path. :frowning: Â Hence why I posted

production environment for this very reason. I’m not convinced that
class

theJRubyClassLoader are leaking. The total number of classes

And the list of next java.lang.ref.Finalizer goes on seemingly

also related:
org.apache.catalina.loader.WebappClassLoader
org.apache.catalina.loader.WebappClassLoader
org.apache.catalina.loader.WebappClassLoader
org.apache.catalina.loader.WebappClassLoader
Jul 20, 2010 8:17:43 AM
Jul 20, 2010 8:17:43 AM
Jul 20, 2010 8:17:43 AM
-Sean

To unsubscribe from this list, please visit:
  http://xircles.codehaus.org/manage_email

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email