Typo's memory growth

marcel · February 26, 2013, 3:56pm

I’m having trouble with Typo growing roughly linearly with the number of
requests, even if the requests are all for the index page. The first
hit results in slightly more growth (~11MB) which is understandable.
But
after that each request increases memory usage by ~2MB, all the way up
to
100 requests.

Here’s a graph of memory usage over number of requests:

http://bit.ly/YkoV0O
(The numbers that went into this graph are below.)

So far, the only tool I have to mitigate this unbounded memory growth is
the PassengerMaxRequests directive.

I see some emails from a long while back about this, but they refer to
blog posts that are no longer available:

Steve L.:
http://rubyforge.org/pipermail/typo-list/2006-August/003228.html
Paul R Brown:
http://rubyforge.org/pipermail/typo-list/2006-August/003293.html

I tried some profiling with memprof. I noticed that several more
ActiveRecord::Relation objects are hanging around after each request.
Each request leaks two more relations for the Article model, for
example.
But I haven’t been able to find what’s holding references to those on
the
heap. Perhaps I’m doing something wrong: First of all, my runtime
object_ids don’t match what memprof dumps in JSON, and second of all,
walking memprof’s JSON heap dump from globals indicates that these
relations are not actually reachable. As I understand, MRI 1.8.7’s
mark-and-sweep GC should free anything that’s unreachable. Not sure why
that doesn’t add up. Here’s how I’m walking the heap:

gist.github.com

https://gist.github.com/mcary/5036169

bfs.rb

require 'progressbar'
require 'shellwords'
require 'json'
require 'active_support/core_ext/hash/keys'
require 'set'
require 'thread' # queue

def main
  objs = load_objs
  $objs = objs

This file has been truncated. show original

Has anyone else:

(a) seen this behavior of memory growth?
(b) figured out how to keep the memory footprint bounded?

When I compare to something like Enki blog, I see that after about two
requests, the memory usage remains constant no matter how many requests
I
make, so it’s not an issue that applies to all Rails apps. (But Typo
has
so many more features than Enki that I would not want to have to
re-implement.)

Thanks,
Marcel

Data from graph linked above (# requests vs. KB RSS Memory):

Trial 1
0 101376
1 112368
2 114540
3 115336
4 116164
5 118564
10 129260
20 147240
30 166068
40 224064
50 241928
60 261136
70 279324
80 296388
90 313772
100 332036

Trial 2
0 101376
1 113856
2 114556
3 116672
4 119772
5 120716
10 130544
20 145788
30 163160
40 224952
50 243680
60 261700
70 280684
80 298232
90 314856
100 334492

marcel · February 26, 2013, 4:13pm

Le 26 fvr. 2013 15:16, marcel [email protected] a crit :

I’m having trouble with Typo growing roughly linearly with the number of
requests, even if the requests are all for the index page. The first hit results
in slightly more growth (~11MB) which is understandable. But after that each
request increases memory usage by ~2MB, all the way up to 100 requests.

Snip

Hi,

Since this list is very low traffic, I’ve created an issue about that
topic on our Github issue tracker. You can find it at

github.com/publify/publify

Typo's memory growth

opened 03:10PM - 26 Feb 13 UTC

closed 09:35PM - 28 Jan 14 UTC

fdv

Copied from Typo mailing list. I'm having trouble with Typo growing roughly lin…early with the number of requests, even if the requests are all for the index page. The first hit results in slightly more growth (~11MB) which is understandable. But after that each request increases memory usage by ~2MB, all the way up to 100 requests. Here's a graph of memory usage over number of requests: http://bit.ly/YkoV0O (The numbers that went into this graph are below.) So far, the only tool I have to mitigate this unbounded memory growth is the PassengerMaxRequests directive. I see some emails from a long while back about this, but they refer to blog posts that are no longer available: - Steve Longdo: http://rubyforge.org/pipermail/typo-list/2006-August/003228.html - Paul R Brown: http://rubyforge.org/pipermail/typo-list/2006-August/003293.html I tried some profiling with memprof. I noticed that several more ActiveRecord::Relation objects are hanging around after each request. Each request leaks two more relations for the Article model, for example. But I haven't been able to find what's holding references to those on the heap. Perhaps I'm doing something wrong: First of all, my runtime object_ids don't match what memprof dumps in JSON, and second of all, walking memprof's JSON heap dump from globals indicates that these relations are not actually reachable. As I understand, MRI 1.8.7's mark-and-sweep GC should free anything that's unreachable. Not sure why that doesn't add up. Here's how I'm walking the heap: https://gist.github.com/mcary/5036169 Has anyone else: (a) seen this behavior of memory growth? (b) figured out how to keep the memory footprint bounded? When I compare to something like Enki blog, I see that after about two requests, the memory usage remains constant no matter how many requests I make, so it's not an issue that applies to all Rails apps. (But Typo has so many more features than Enki that I would not want to have to re-implement.) Thanks, Marcel Data from graph linked above (# requests vs. KB RSS Memory): Trial 1 0 101376 1 112368 2 114540 3 115336 4 116164 5 118564 10 129260 20 147240 30 166068 40 224064 50 241928 60 261136 70 279324 80 296388 90 313772 100 332036 Trial 2 0 101376 1 113856 2 114556 3 116672 4 119772 5 120716 10 130544 20 145788 30 163160 40 224952 50 243680 60 261700 70 280684 80 298232 90 314856 100 334492

Best regards,
Fred