Jruby-1.7.3 with rails 2.3.17 on tomcat giving timeout issues

Ruby JRuby
1

Hi All,

We are using jruby-1.7.3 with rails 2.3.17. Running this on tomcat
server in the production environment.We are running it on two tomcat
nodes.

When load goes high on production environment. We are seeing lot of
timeout errors and server response time goes high. Also, CPU utilization
spikes out.

Just wanted to check if something seriously wrong with jruby-1.7.3 with
respect to time out. Or is there any issue in running the parallel
threads.

Thanks in Advance for help !!

2

Hey, that’s very likely not a JRuby issue (although there should be no
blocker for you guys to simply update to the latest of 1.7.x and see if
it’s the same) but an I/O one.

Something is waiting for IO (a Ruby library/gem such as memcached or
it’s
alternatives or a low pool setting) - you need to review your gems or
even
better monitor what’s going on.

K.

3

I would tend to agree with Karol. Is there a stack trace in the logs
that would clue you into where the timeout is happening in the code?

Thanks,
Ariel

Sent from my mobile device. Please excuse any errors.

4

Thanks Karol and Ariel,

These are random stack traces which we are getting upon hitting the
application multiple times.

The exact same code base and gems were working fine on our older stack
which was JBOSS and Jruby-1.4.x and Rails - 2.3.17

Status: 500 Internal Server Error
could not obtain a database connection within 5 seconds. The max pool
size is currently 90; consider increasing it.

Status: 500 Internal Server Error
Detected invalid hash contents due to unsynchronized modifications
with concurrent users

RuntimeError (can’t add a new key into hash during iteration):
org/jruby/RubyHash.java:946:in []=' org/jruby/RubyProc.java:261:incall’
org/jruby/RubyProc.java:209:in `call’

So am thinking that it might be issue with jruby thread-safety

Any pointers will be really helpful.

5

I do recall there being many issues with rails 2.x around thread safety.
I assume that your application is running with “config.threadsafe!”?

Does the concurrent modification error also happen due to connection
pool errors?

It looks like there are not enough connections in the pool to satisfy
the number of concurrent requests tomcat is dispatching and/or some
requests take too long, which causes other threads to timeout.
Have you tried increasing the checkout_timeout or the number of threads?

What version of JBoss are you using where everything seems to work fine?
If I recall correctly it uses tomcat as the servlet container. Are the
versions and configuration the same (eg maxSpareThreads)

Thanks,
Ariel

Sent from my mobile device. Please excuse any errors.

6

Adding upon Ariel’s suggestions - the concurrency issue your pointing
out
with the Hash is something that should be the first thing to look at.

Unfortunately we do not see the full trace … I’m guessing your
app/code
is not really thread-safe … working on an old version of JRuby + JBoss
does not mean it was all fine.
Maybe you’ve run with JRuby-Rack and it pools runtimes by default ? On
the
other hand Rails 2.3 is not 100% (depends on it’s usage) thread-safe
e.g.
I’ve seen issues with the 2.3 connection pool not behaving consistently
when a connection is closed and re-required on the same thread (might
happen with the active_record session store).

I might be wrong in the end but do not expect to get all of this worked
out
if you’re just shooting (helped some and their issues were mostly
non-trivial - required understanding of the app on the level of user
code
or at least knowing the configuration/gems used) - it’s best to get
someone
to look at the app or simply start with the “easy” fixes one thing at a
time.

K.

7

A little snooping around rails defects it appears to be fixed in 3.2

Thanks,
Ariel

Sent from my mobile device. Please excuse any errors.

8

I suggest you take a thread dump and check the java heap use and GC
also.

Thanks
Alex

9

I’m not sure it’s only that - have tried to track down comparing against
3.2, but it’s not straight-forward since code got “re-invented”
(refactored) …
I’ve ended up carefully back-porting AR’s 3.2 pool with AR-JDBC 1.3.x
for
Rails 2.3 and it worked without any (production) issues.
The other thing we should have noted as well and I certainly recommend (
although I’m biased :slight_smile: is to upgrade AR-JDBC to 1.3.x since some parts
previously not completely AR compatible might cause issues with some
adapters as well (e.g. reconnect!).

K.