(ultra)sphinx reindexing eats up a lot of memory

Hello,

On my 512MB slice (Slicehost) running 3 thins and a Postgres database, I
had set up a delta indexing scheme that executes “rake ultrasphinx
index:delta” task every minute via cron. The server was unstable
(database process got killed every day or so). I isolated the problem to
that cron task (eating up too much memory). After temporarily disabling
it, the server was stable.

I wonder if some of you also had similar problems and if the solution is
to simply use a bigger slice…

Thank you!
david

I haven’t used the UltraSphinx plugin, but Sphinx itself has a mem_limit
configuration in its conf file to cap the amount of RAM that the indexer
will use.

Roberto
blog.idearise.com

On Tue, Sep 23, 2008 at 10:11 PM, D. Krmpotic <

Roberto, thank you for the answer.

I checked and saw that I already have reduced mem_limit for sphinx
indexer to 64MB.
It seems that even this is too much for my slice, when it shouldn’t
be… Check this (I’m baffled):

Mem: 523008k total, 470204k used, 52804k free, 1896k buffers
Swap: 0k total, 0k used, 0k free, 41524k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2218 david 15 0 204m 108m 1936 S 0 21.2 2:22.51 thin
2223 david 15 0 201m 104m 2100 S 0 20.5 1:33.93 thin
2210 david 15 0 130m 60m 1352 S 0 11.8 0:04.31 thin
13651 david 16 0 108m 34m 1464 S 0 6.8 0:01.52 ruby
2102 mysql 15 0 162m 32m 1524 S 0 6.4 0:07.48 mysqld
2378 root 16 0 40608 6512 856 S 0 1.2 0:02.34 munin-node
2472 postgres 16 0 99.6m 4836 2168 S 0 0.9 0:00.85 postgres
13653 postgres 16 0 99.1m 4064 1876 S 0 0.8 0:00.03 postgres
11600 david 16 0 19760 3384 940 S 0 0.6 0:00.12 bash
2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres

So out of 512MB, only 52MB is free and that’s why indexer reported
“Cannot allocate memory”.

Can you please help me understand how can 90% of memory be used if first
5 lines in TOP total to 63% (the rest is negligible). So 3 thins, nginx,
mysql and postgres should not consume 90% of 512MB and the percentages
in the table confirm this. But still I only have 52MB left (!?).

Thank you for any clarifications.
David

Roberto R wrote:

I haven’t used the UltraSphinx plugin, but Sphinx itself has a mem_limit
configuration in its conf file to cap the amount of RAM that the indexer
will use.

Roberto
blog.idearise.com

On Tue, Sep 23, 2008 at 10:11 PM, D. Krmpotic <

I don’t know anything about Slicehost - but surely you should have
some swap space?

Cheers,
Tony

2008/9/24 D. Krmpotic [email protected]:

2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres

blog.idearise.com


Tony White

Yes, that’s true (and strange). It should start swapping if nothing
else… but still… swapping is just as bad … and 512MB should be
enough for my needs… the problem is finding out what is eating the
rest of the memory.

BTW - free reports this:

free -m
total used free shared buffers
cached
Mem: 510 470 40 0 11
45
-/+ buffers/cache: 413 97
Swap: 0 0 0

This is quite a problem for me and I hope someone can suggest a
solution! Thanx!

david

Tony White wrote:

I don’t know anything about Slicehost - but surely you should have
some swap space?

Cheers,
Tony

2008/9/24 D. Krmpotic [email protected]:

2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres

blog.idearise.com


Tony White

Hi,

It looks like there is no swap (???).

On Wed, Sep 24, 2008 at 12:30 PM, D. Krmpotic <

D. Krmpotic wrote:

free -m
total used free shared buffers
cached
Mem: 510 470 40 0 11
45
-/+ buffers/cache: 413 97
Swap: 0 0 0

This is quite a problem for me and I hope someone can suggest a
solution! Thanx!

Linux aggressively caches files using available free memory (and even
sometimes swapping out lesser used programs for more cache),
consequently “free” memory is not just the reported unused memory, but
unused + buffers + cache. So, when you looked at top, it said 52M free,
but the effective free memory is 97M as reported by that second line
from free.

Your VM does appear to have swap disabled, which can be a good idea if
you have a very well tuned and controlled memory profile, but quite a
problem if not. Yes it’s best never to swap, but it might be preferable
to go slow instead of fail completely when you run out.

Can you please help me understand how can 90% of memory be used if
first
5 lines in TOP total to 63% (the rest is negligible). So 3 thins,
nginx,
mysql and postgres should not consume 90% of 512MB and the
percentages
in the table confirm this. But still I only have 52MB left (!?).

Well, as we just discussed, you actually have about 80% used, not 90%,
when you reclaim the OSes cache slush, Don’t discount all those little
processes – you likely have several dozen to a hundred 1-5M processes
below your top five – it adds up.

Looking back at your earlier top output, I also notice you have Pg
running with about 90M shared buffers (inferred from the virtual memory
size), but it normally uses only 2-4M RSS:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2472 postgres 16 0 99.6m 4836 2168 S 0 0.9 0:00.85 postgres
13653 postgres 16 0 99.1m 4064 1876 S 0 0.8 0:00.03 postgres
2476 postgres 16 0 99.9m 3356 812 S 0 0.6 0:00.39 postgres

It’s possible that the sphinx indexing operation, which queries postgres
directly, causes Pg to temporarily allocate substantially more memory
then the normal working set to serve the re-index requests, and that
could be another source of your memory over-commit.

If your principle database is postgres, I gotta wonder why you have a
32M mysqld running too?

2102 mysql 15 0 162m 32m 1524 S 0 6.4 0:07.48 mysqld

-Devin

Yeah, without any swap… you’re going to have some problems.
Surprised that you don’t have any enabled by default. (at least 2x the
amount of RAM to start… )

On Wed, Sep 24, 2008 at 9:02 AM, Roberto R [email protected]
wrote:

else… but still… swapping is just as bad … and 512MB should be
-/+ buffers/cache: 413 97

some swap space?


Robby R.
Chief Evangelist, Partner

PLANET ARGON, LLC
design // development // hosting

http://www.robbyonrails.com/
aim: planetargon

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4068 [fax]

in the meantime I did enable it… no idea how it got disabled…

but still… do you think it’s ok that so little memory is left on boot
and reindexing would have to start swapping?

Robby R. wrote:

Yeah, without any swap… you’re going to have some problems.
Surprised that you don’t have any enabled by default. (at least 2x the
amount of RAM to start… )

On Wed, Sep 24, 2008 at 9:02 AM, Roberto R [email protected]
wrote:

else… but still… swapping is just as bad … and 512MB should be
-/+ buffers/cache: 413 97

some swap space?


Robby R.
Chief Evangelist, Partner

PLANET ARGON, LLC
design // development // hosting

http://www.planetargon.com/
http://www.robbyonrails.com/
aim: planetargon

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4068 [fax]

for those interested how swap got disabled… it was probably during an
upgrade to 8.04 (Ubuntu).

I enabled the swap partition with SWAPON command, but discovered that
upon reboot it got disabled again…

I found out the problem (UUIDs for sda2 in /etc/fstab were wrong…):

:~$ cat /etc/fstab
proc /proc proc defaults 0 0

/dev/sda1 – converted during upgrade to edgy

UUID=9c65b16b-2082-4121-9c78-d5fc6e407ade / ext3
defaults,errors=remount-ro 0 1

/dev/sda2 – converted during upgrade to edgy

UUID=ff26b2f9-e08c-42d8-97fa-9a5e37f9a711 none swap sw 0 0

The correct UUID:

:~$ cat /etc/blkid.tab
/dev/sda1
/dev/sda2

So I updated the UUID in fstab and the swap partition settings survive
the reboot now!

david

D. Krmpotic wrote:

in the meantime I did enable it… no idea how it got disabled…

but still… do you think it’s ok that so little memory is left on boot
and reindexing would have to start swapping?

Robby R. wrote:

Yeah, without any swap… you’re going to have some problems.
Surprised that you don’t have any enabled by default. (at least 2x the
amount of RAM to start… )

On Wed, Sep 24, 2008 at 9:02 AM, Roberto R [email protected]
wrote:

else… but still… swapping is just as bad … and 512MB should be
-/+ buffers/cache: 413 97

some swap space?


Robby R.
Chief Evangelist, Partner

PLANET ARGON, LLC
design // development // hosting

http://www.planetargon.com/
http://www.robbyonrails.com/
aim: planetargon

+1 503 445 2457
+1 877 55 ARGON [toll free]
+1 815 642 4068 [fax]

I would like to know how to use cron with sphinx. I would be kind
enough to assist me?

Thanks,

Emeka