Memory leak issue after restart of rails application in tomcat

Hello,

I am debugging a memory leak issue after the restart of my rails
application
in tomcat. I am using tomcat manager to stop and start the application.
Memory usage goes up after every restart and it becomes unusable after
few
restarts. It is the only ruby/rails application that is deployed in
tomcat.
When I used jstack to list java threads, I noticed that there are many
jruby
threads running even after the application has been stopped. Number of
threads keeps growing after each restart.

I did try the following code to kill the threads:
at_exit do
Thread.list.each { |t| Rails.logger.warn { “Thread Status is:
#{t.inspect}, #{t[:name]}, #{t.status}” } }
Thread.list.each { |t| t.exit unless t == Thread.current }
Rails.logger.warn { “EXITING**” }
end

2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0x99dde9sleep, synchronize_users, sleep
2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0x1b80d21sleep, owner_audit, sleep
2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0x660d14run, , run
2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0x1e4c027run, , run
2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0x1036c7fsleep, , sleep
2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0x1c814basleep, flow_audit, sleep
2010-07-27 17:46:41,531 WARN ( ?:?) - Thread Status is:
#Thread:0xc8e634run, , run
2010-07-27 17:46:41,546 WARN ( ?:?) -
EXITING**

However, I still see lots of jruby threads in tomcat after stopping the
application and memory issue.

Here is my environment:
OS: Windows 2003
Jruby: 1.5.0
Tomcat: 6.0.18

Any idea how to resolve this issue?

Thanks,
-Rakesh

I am still trying to figure out the root cause of the memory leak during
stop/restart/redeploy of application in tomcat.

I have very simple test rails application deployed in tomcat and
analyzing
it using jconsole. I noticed that the number of classes goes up by about
3000 and memory usage goes up by about 40M for every restart. Also
noticed
that every restart adds a new jruby runtime entry in jconsole
org.jruby->runtime (under MBean tab), so the jruby runtimes keeps
growing. I
have the following in my application’s web.xml file:

jruby.max.runtimes
1

jruby.min.runtimes 0

is this an issue? Any idea how I can fix this?

Thanks,
-Rakesh

I am now using the latest jruby (snapshot) and trying to debug this
issue.

  • I have set jruby.max.runtimes and jruby.min.runtimes to 1 in web.xml.
    However, I see two instances of runtime in jconsole. Is that an issue?
  • After undeploying the app in tomcat, one instance of runtime is
    removed
    but one instance remains. I confirmed that org.jruby.Ruby.tearDown is
    only
    called once.
  • If I don’t set the jruby.max.runtimes and jruby.main.runtime in
    web.xml
    the there is only instance of runtime. However, it remains there after
    undeployment. I confimed that org.jruby.Ruby.tearDown is not called at
    all.

Is that Rack issue?

Thanks,
-Rakesh

Carl,

Last two posts are updates to the original post. I am trying to debug
the
issue of classes being not unloaded during undeployment of jruby rails
application in tomcat. I am posting my findings to facilitate if someone
decides to jump in and help.

-Rakesh

Dude, let us absorb the first question before you ask 10 more.

Hope that helps!
Carl

Rakesh,

You might be seeing the effects of this bug:

http://jira.codehaus.org/browse/JRUBY-4981

To see if it helps, you can either build yourself a trunk snapshot,
wait for one to be built tonight by CI
(http://ci.jruby.org/snapshots), or wait for the 1.5.2 release which
will have the fix.

Cheers,
/Nick

On Tue, Aug 3, 2010 at 1:45 PM, Rakesh A. [email protected] wrote:

wrote:

To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

You might want to do what I recommend in my post here to get and
analyze a heap dump:

Specifically, use jmap to get a dump and jhat to poke around it, or
put the dump somewhere “we” can have a look.

You’ll want to do it like this:

  1. deploy your app
  2. hit your app sufficiently enough to cause the cross-deploy leaking
    you’re seeing
  3. undeploy your app
  4. pull the dump (and possibly use VisualVM, also described in the
    above post, to force a few GC runs)

If the app is “fully” undeployed, we should not see any Ruby objects
in flight. If we see Ruby objects in flight, we’ll see entire JRuby
runtimes in flight, and can track why they’re still in memory.

FWIW, there are almost no Java app servers that don’t leak to some
degree across deploys, and most of the time it’s not our fault. JDBC
drivers can kill us, globally-registered services and caches can kill
us, and so on. But we haven’t investigated in a long time…so it
would be worth having you pull a dump and investigating.

  • Charlie

On Fri, Jul 30, 2010 at 2:03 PM, Rakesh A. [email protected] wrote:

I have very simple test rails application deployed in tomcat and analyzing
Â
On Tue, Jul 27, 2010 at 5:07 PM, Rakesh A. [email protected] wrote:

2010-07-27 17:46:41,531 WARNÂ (Â Â Â Â Â Â Â Â Â Â Â ?:?)Â Â Â Â - Thread Status is:
#Thread:0xc8e634run, , run

Any idea how to resolve this issue?

Thanks,
-Rakesh


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

On Wed, Aug 4, 2010 at 3:38 PM, Rakesh A. [email protected] wrote:

I got the jruby jar from the last night build for my test application and
still seeing jruby classes after undeploying the application and manually
kicking the GC.

Charles: Thanks for the link. I have being trying to analyze the heap
dump,but haven’t got the root cause yet. I tried the Leak Suspect report of
Eclipse Map and it is reporting:

If you can post your test application we can also try to look for the
problem on this end. Basically, you’ll want to pick one of the
org.jruby.Ruby instances (look at the object histogram view) and from
its
context menu choose “Show paths to GC roots”…“Exclude all phantom,
weak,
soft, etc references”. The resulting view will show why JRuby
instances
are not getting collected when you undeploy.

It’s a bit of a challenge to investigate these sorts of things, but it’s
an
interesting adventure. Get that demo app posted (probably to a bug) and
we’ll help.

I got the jruby jar from the last night build for my test application
and
still seeing jruby classes after undeploying the application and
manually
kicking the GC.

Charles: Thanks for the link. I have being trying to analyze the heap
dump,but haven’t got the root cause yet. I tried the Leak Suspect report
of
Eclipse Map and it is reporting:
Problem Suspect 1

1,051 instances of “org.jruby.RubyClass”, loaded by
“org.apache.catalina.loader.WebappClassLoader
@ 0x84f94180”
occupy 8,255,704 (27.46%) bytes.

Biggest instances:

  • org.jruby.RubyClass @ 0x85fdf9a0 - 474,208 (1.58%) bytes.

Keywords
org.apache.catalina.loader.WebappClassLoader @ 0x84f94180
org.jruby.RubyClass
Details » <file:///tmp/report4308993701297866403/pages/20.html>
<file:///tmp/report4308993701297866403/index.html#> Problem Suspect 2

557 instances of “org.jruby.RubyModule”, loaded by
“org.apache.catalina.loader.WebappClassLoader
@ 0x84f94180”
occupy 4,580,576 (15.24%) bytes.

Biggest instances:

  • org.jruby.RubyModule @ 0x8506e578 - 619,512 (2.06%) bytes.

These instances are referenced from one instance of “org.jruby.Ruby”,
loaded by “org.apache.catalina.loader.WebappClassLoader @ 0x84f94180”

Keywords
org.jruby.Ruby
org.apache.catalina.loader.WebappClassLoader @ 0x84f94180
org.jruby.RubyModule
Details » <file:///tmp/report4308993701297866403/pages/22.html>
<file:///tmp/report4308993701297866403/index.html#> Problem Suspect 3

1,495 instances of “org.jruby.MetaClass”, loaded by
“org.apache.catalina.loader.WebappClassLoader
@ 0x84f94180”
occupy 3,663,568 (12.19%) bytes.

Biggest instances:

  • org.jruby.MetaClass @ 0x85c42788 - 332,584 (1.11%) bytes.
  • org.jruby.MetaClass @ 0x8580e948 - 311,616 (1.04%) bytes.

Keywords
org.jruby.MetaClass
org.apache.catalina.loader.WebappClassLoader @ 0x84f94180

On Wed, Aug 4, 2010 at 1:05 AM, Charles Oliver N.

The issue reported in this jira is exactly what I see with jruby 1.5.1
and
1.6.0.dev. I tried with the memtest application and classes are not
unloaded
at redeployment.
http://jira.codehaus.org/browse/JRUBY-3281

I had to clone the jira to attach the jconsole screen shot.
http://jira.codehaus.org/browse/JRUBY-4996

Can someone please update the Affects Version field.

Thanks,
-Rakesh

On Thu, Aug 5, 2010 at 3:34 PM, Charles Oliver N.

Thanks Charlie,

I have attached the test application to the jira.

-Rakesh

On Sat, Aug 7, 2010 at 3:02 AM, Charles Oliver N.

Ok, let’s give it another shot. If you can provide a sample app, we’ll
take
a look at it.

I commented and updated the affects version.

  • Charlie