Concurrency issues with JRuby on Rails

Ruby JRuby
1

Hello there,

I’m running into some concurrency issues and need some help. I’m
currently
using JRuby 1.7.12 and Rails 4.1.1. Every now and then i’m getting this
error:

ActiveRecord::ConnectionTimeoutError (could not obtain a database
connection within 15.000 seconds (waited 15.001 seconds)):

vendor/bundle/jruby/1.9/gems/activerecord-4.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:190:in
`wait_poll’

Our database pool is set to 20 and a checkout_timeout of 15.

Has anyone seen this issue before?

2

Hey Bruce,

Thanks for the quick response.

  1. The rails server is only taking 2-5 connections at this point in
    time.
    We’re testing atm.
    • No there are not many concurrent connections.
    • The requests do take a significant amount of time 20+ seconds.
      (Returning json data hitting many different apis)
  2. I am not aware of any AR calls outside of the typical rails
    controller
    actions.
    • There may be a db leak but how do I tell? Is there a variable, etc
      I
      can debug to print that info out? Remaining connections perhaps?
  3. I could increase the pool higher but with the info above there is
    most
    likely a leak or something going wrong with the connection pool.

We are also using threading in the action that is causing the issue. But
we
have used the “with_connection” block before making any database calls
in
the thread.

3

It sounds like the twenty database connection you are allowing are all
in use for more than 15 seconds.

Is your Rails server under much load? (Are there many, concurrent
requests? Are a bunch of those requests taking a long time (several
seconds) to complete?)

Is there a chance that your application is leaking database connections?
That is, might their be code paths where a connection is taken from the
database connection pool, but never returned to the pool?

Can you increase the size of your database connection pool?

  • Bruce

P.S. For what it’s worth, I recommend a low checkout_timeout,
specifically one second. I don’t what a user request waiting around
for resources. Fifteen seconds is a very long time for a user to
wait for a web request. I’d rather give the user (and my own error
logs) a message quickly when I run out of resources. With an error
message, I get earlier notice that something isn’t quite right.

4

Chason -

Interesting. I’ve done some fun stuff with concurrency under Rails, but
my stuff didn’t deal with database connections in threads. So, I don’t
don’t have direct experience to help you here. This (long) blog post,
ActiveRecord Concurrency in Rails4: Avoid leaked connections! – Bibliographic Wilderness,
looks like it might be relevant to what you are seeing.

  • Bruce