I have a JRuby on Rails app that was not able to handle a production
load of ~300 requests/min, and I’d like some direction troubleshooting
it. I know of several things that are inefficient, but need to figure
out what the best bang-for-the buck would be.
Conceptually it’s a simple app: It receives a search term, connects to
a Google Custom Search account, receives an XML response, parses it,
and displays the results. A second AJAX request pops in a sidebar with
related items queried from a database. The first request takes the
most time.
It’s Rails 2.3.5 and JRuby 1.5.2 being deployed as a .war to a single
instance of Tomcat 6 on a Sun T5250/Solaris with 16 CPUs and 256GB
RAM, though this is shared by many other apps, some in separate
containers (VMs).
A few minutes under load and it was consuming 8GB(!) of RAM and
running extremely slowly. The page was taking over 30 seconds to
appear, but the in-app timers and New Relic were both showing “normal”
execution times of a little over 1 second, as if there was some
queuing going on or something outside the controller holding things
up.
Suspected inefficiencies:
- I know I should bring it up to JRuby 1.5.6 at least, and I’m sure
1.6.0 will have additional improvements. I’m sure there may also be
improvements to JRuby-Rack and Warbler since we first incorporated
them into the project. Is there anything in the newer stack likely to
make a huge difference?
- Warbler is configured as threadsafe but the runtimes are set to min/
max of 2 and 4. New Relic is showing 4 runtimes. I haven’t found much
guidance for these settings (especially the interaction of threadsafe
with runtimes), and am not sure if setting them to 1 and 1 would
improve things? Or should they go higher? What’s the downside to
setting this to a very large number?
- The memory consumption concerns me, and I’m wondering if it’s maxing
out at the measured 2GB/runtime. The app uses REXML (actually JREXML,
but it doesn’t seem to make a difference) and I have a feeling this
should be swapped out immediately with Nokogiri or a Java XML parser.
- We’ve found that the Sun doesn’t have a lot of raw speed but should
scale out well IF I’m taking advantage of threading and multiple CPUs.
Should I be using multiple Tomcat instances and balance between them
with Apache?
Any guidance toward the most obvious problems, or additional things I
haven’t thought of, would be appreciated.
Hi Mark -
I’ve never had to deal with a load of 300 req/min, but I’ve deployed
apps
used by around 400 intranet users, doing some heavy database work for an
internal financial/employee management intranet.
Here are some general things I have found, which I hope help you:
-
Over the last couple years, I have deployed in both windows/linux and
for
tomcat 5.5 and 6.0. I believe my very first deployment was with jruby
1.2.
Deployment is getting better and better with each version.
-
In my experience, using memcache with the jruby memcache client gem (
GitHub - ikai/jruby-memcache-client: A wrapper of the Java Whalin MemCache client for JRuby) offers the best session
performance. For whatever reason, I’ve had very poor results with AR
sessions.
-
if you haven’t, increase Xms and Xmx to adequate levels, and force
jruby
to compile (other settings here:
http://kenai.com/projects/jruby/pages/PerformanceTuning)
-
I do not deploy anything else in tomcat except a single jRoR
application.
If I need to deploy multiple apps, they each get their own tomcat
instance
-
I have noticed that given enough time, my applications always run out
of
PermGen space or throw an OutOfMemoryException. Because of this, I often
schedule a tomcat restart once per day, early in the morning. Obviously
this
causes downtime. In my situation, it is acceptable since it is an
intranet
environment, and typically during business hours only.
-
Question for you: What library are you using to connect to Google?
I’ve
had success with ruby-net-ldap, but again, I’ve never had to deal with a
high load. I think it would be worth investigating whether or not the
library you are using can handle a high load. (or are you issuing these
requests on the client end via javascript/ajax?)
Hope this helps,
Jin L.
- Make sure that you have config.threadsafe! in production.rb
- Multi threaded should have only 1 runtime running.
so make sure you comment out the min and max runtimes in your
warbler.rb.
The config.threadsafe! configuration indicates to warbler to use only 1
runtime by default.
- The default heap/memory size for java is generally not enough when
running rails app using jruby. Here are the setting
I have in my catalina.sh file which starts tomcat:
-server option for better optimization
JAVA_OPTS=“$JAVA_OPTS -server”
jvm needs more memory to run jruby
CATALINA_OPTS=“$CATALINA_OPTS -Xmn128m -Xms256m -Xmx256m”
Mark T.-5 wrote:
most time.
queuing going on or something outside the controller holding things
guidance for these settings (especially the interaction of threadsafe
with Apache?
–
View this message in context:
http://old.nabble.com/Help-me-troubleshoot-a-poorly-performing-JRoR-app-tp30888628p30889443.html
Sent from the JRuby - User mailing list archive at Nabble.com.
On Feb 9, 11:56pm, ajosephmi [email protected] wrote:
- Make sure that you have config.threadsafe! in production.rb
- Multi threaded should have only 1 runtime running.
so make sure you comment out the min and max runtimes in your warbler.rb.
The config.threadsafe! configuration indicates to warbler to use only 1
runtime by default.
So if I have config.threadsafe! and min/max at 2 and 4 it is not
running in multithreaded mode?
Are there any circumstances where I’d benefit from a large value for
the max runtimes?
No you cannot have multithreaded with multiple runtimes . Its one or the
other.
With your configuration your app will either run a single runtime
handling
multiple requests or multiple runtimes each handling one request at a
time
because your configuration is in conflict. Looks like from your memory
profile warbler is running multiple runtimes with each runtime handling
a
single request at a time.
You can have multiple installations of tomcat running your app in multi
threaded mode with a load balancer in front of it. Off course now the
architecture is different.
For rails 2.2.2 onwards multi threaded setup (single runtime) is the
recommended setup.
Jruby/tomcat should easily handle the load you have. May be there is a
memory leak which could be due to some gems you are using or jruby
itself.
Mark T.-5 wrote:
–
View this message in context:
http://old.nabble.com/Help-me-troubleshoot-a-poorly-performing-JRoR-app-tp30888628p30892093.html
Sent from the JRuby - User mailing list archive at Nabble.com.
- Question for you: What library are you using to connect to Google? I’ve
had success with ruby-net-ldap, but again, I’ve never had to deal with a
high load. I think it would be worth investigating whether or not the
library you are using can handle a high load. (or are you issuing these
requests on the client end via javascript/ajax?)
It’s plain ol’ Net::HTTP. I did at one point in the development swap
in JRuby-RestClient, a wrapper around the Java Apache Commons
HttpClient library as an experiment. This change was not tested under
load. I went back to Net::HTTP because it was easier to mock in my
unit tests.
Mark,
I used to be on a project running JRuby on Solaris and we had
experienced
this environment as fairly inferior in performance to Linux, even with
the
general performance tweaks that are commonly mentioned (e.g., client vs.
server vm – see here: http://www.javaworld.com/community/node/4113 ).
You didn’t say whether you were doing any database access or not, but
one of
our Solaris bottlenecks was that we were using SQLite. The SQLite JDBC
driver ships with native binaries for Windows, Linux, and Mac OS X, but
not
Solaris. If you don’t compile the native binaries yourself for Solaris,
SQLite JDBC falls back on the 100% pure Java driver, which is a lot
slower,
at least on Solaris.
Sean L. wrote in post #980860:
How can someone tell if config.threadsafe! is working correctly?
I commented out min/max in warbler.rb and uncommented config.threadsafe!
in
the environments where I wanted it turned on. But the logs don’t clearly
state that I’m running in threadsafe mode and, in fact, it looks like it
launches two runtimes anyway:
INFO: Info: add application to the pool. size now = 1
INFO: Info: add application to the pool. size now = 2
INFO: Info: using runtime pool timeout of 30 seconds
INFO: Warning: no min runtimes specified.
INFO: Warning: no max runtimes specified.
Yet it’s performing pretty snappy. Any suggestions?
-Sean
The warning about “no min/max runtimes specified” is first sign of a
possible memory utilization problem due to “unlimited” runtimes as load
increases. Though you have uncommented config.threadsafe!, suggest you
to also set config.webxml.jruby.min.runtimes and
config.webxml.jruby.max.runtimes to “1” in warbler.rb. This is how we
contained the memory utilization by jruby runtimes. You can verify if
(min and max runtime = 1) + (config.threadsafe!) is working or not by
using profiler like jvisualvm (we used it successfully for the same).
Note the number of classes loaded shown in left bottom quadrant of the
jvisualvm dashboard. If the number is constant around 8K (~ classes
required for 1 runtime in JRuby 1.5.0) and doesn’t increase considerably
with load, then you have achieved what you wanted.
How can someone tell if config.threadsafe! is working correctly?
I commented out min/max in warbler.rb and uncommented config.threadsafe!
in
the environments where I wanted it turned on. But the logs don’t clearly
state that I’m running in threadsafe mode and, in fact, it looks like it
launches two runtimes anyway:
INFO: Info: add application to the pool. size now = 1
INFO: Info: add application to the pool. size now = 2
INFO: Info: using runtime pool timeout of 30 seconds
INFO: Warning: no min runtimes specified.
INFO: Warning: no max runtimes specified.
Yet it’s performing pretty snappy. Any suggestions?
-Sean