Hoping for guidance here…
We are running a Sinatra app under the the Glassfish gem. I have remote
processes that send it requests every 10 seconds - so in a given minute
I
see 6 requests per remote process. It takes < .5 second to handle these
requests.
As the remote processes go up, the app starts crawling. I find that with
merely 5 remote processes connecting, the log shows them being processed
every 23 seconds, not 10!
If I up the jruby-runtime-pools to a min/max of 5 or so, everything
works
fine again with the requests being processed at their incoming rate of
10
seconds.
I don’t understand why these very quick requests are processed so
slowly. I
would hope we could handle hundreds of such devices (without resorting
to
giant jruby-runtime-pool values).
How might I debug this? Jconsole shows nothing close to max capacity
(or
rising toward it). It appears that everything from rack down is snappy,
but
the requests themselves appear ‘queued’ in some way. Maybe I should try
the
glassfish server instead of the gem?
If it looks like the ruby stuff is responding quickly… you might try
playing with YourKit (a profiler) to see if you can discern anything?
I’ve used YourKit very successfully to uncover memory leaks. But it
also provides profiling capabilities.
If by upping runtimes you are getting better results… it could be
things are not being processed in a multithreaded manner… but I
would think if it takes .5 seconds to process a request you’d be
getting better responses with 1 runtime.
Sorry if this doesn’t help much… but without seeing any code
examples it’s the best I Can come up with.
On Mon, Mar 8, 2010 at 10:12 PM, Ken at Inbox256 [email protected]
wrote:
the requests themselves appear ‘queued’ in some way. Maybe I should try the
glassfish server instead of the gem?
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
Can you load your app w/ mongrel instead of glassfish to see if that
makes a difference. We could nail down if it is glassfish or not.
also provides profiling capabilities.
On Mon, Mar 8, 2010 at 10:12 PM, Ken at Inbox256 [email protected]
seconds.
snappy, but
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
On Tue, Mar 9, 2010 at 8:11 AM, Jay McGaffigan [email protected]
wrote:
If by upping runtimes you are getting better results… it could be
things are not being processed in a multithreaded manner… but I
would think if it takes .5 seconds to process a request you’d be
getting better responses with 1 runtime.
Yeah, this was my thought too. It sounds like there’s some contention
in the GF gem or in the app that’s preventing it from handling
requests more quickly. A Sinatra app should probably run fine
“threadsafe” with a single runtime handling all requests, so it’s
peculiar that it improves with more runtimes.
And we do know of other users running GF gem + Sinatra for really
high load applications, in the neighborhood of thousands of requests
per second. So something’s bogging this particular case down.
Ken: So the problems seem to arise when you have many concurrent
clients requesting things, yes? There is another possibility…it
could be spending all its time in GC. Perhaps you could run JRuby + GF
gem like this and see if the GC times get really bad:
jruby -J-verbose:gc -S glassfish …
If the times get really high (especially for the “Full” collections)
it’s possible your application is generating too much garbage per
request. Large amounts of garbage can be handled pretty efficiently by
one thread, but when you have many generating that much garbage, the
system can start spending all its time GCing.
Try both of these separately…
- Try to get GF running “threadsafe” for your Sinatra app (perhaps
min/max runtimes = 1?) and see how it looks.
- Get some GC results and examine your app usage to see if it’s
creating too much garbage (like loading a giant object graph and
throwing it away every request)
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
Grizzly starts with max 5 threads in the thread pool serving requests in
the
queue. See glassfish.yml (gfrake config) and you can tune the grizzly
threadpool as well as request-timeout settings.
to fix such long-held http requests, glassfish gem should provide
continuation (comet/async support) so that the http worked thread is
released immediately.
-vivek.
Guys, thanks for the updates. I believe the problem is related to 64
bit
windows. I cannot recreate on 32 bit systems. So it is either win2008
or
64 bit, and I’m guessing the latter.
I used the default jruby install. Perhaps I need to manually grab a 64
bit
jre?
So it runs fast on 32-bit? That’s rather peculiar…the only things
that would really be different under 64-bit would be the native
libraries. Perhaps there’s some odd bottleneck in one of them?
Have you been able to narrow it down to a simpler case or to a
specific API or call in your application? Perhaps you can get
request-processing and most of Rails out of the way?
On Thu, Mar 11, 2010 at 11:35 AM, Ken at Inbox256 [email protected]
wrote:
Grizzly starts with max 5 threads in the thread pool serving requests in
Hoping for guidance here…
If I up the jruby-runtime-pools to a min/max of 5 or so, everything works
glassfish server instead of the gem?
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
Sorry, you’re not running Rails…I meant get all request-processing
and (Sinatra) out of the way so we can start to narrow
down what’s actually slow.
On Mon, Mar 15, 2010 at 10:55 AM, Charles Oliver N.
[email protected] wrote:
windows. I cannot recreate on 32 bit systems. So it is either win2008 or
threadpool as well as request-timeout settings.
We are running a Sinatra app under the the Glassfish gem. I have remote
seconds.
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email