Ruby Forum NGINX > Memcached module -- unix domain socket support? (too many TIME_WAITs..)

I have a pool of memcached servers running on localhost with
connection to nginx, feeding motion jpeg data out via static JPEGs to
a flash application. The load is pretty high on the server because of
this with each client running 3 GETs from memcached every second (i.e.
3fps).

Nginx works fine, however, the problem I have is that thousands of
TIME_WAITs are created from the memcached GET requests, which I would
like to eliminate. I've already tuned the kernel but this has not
reduced it enough to handle a couple thousand viewers on the server.

Is there any way to connect nginx's memcached module to memcached via
unix domain socket or perhaps UDP, or any other ideas on how to fix
this problem?

Cheers
Kon

On Thu, Jul 31, 2008 at 09:54:03PM -0700, Kon Wilms wrote:

> 
> Is there any way to connect nginx's memcached module to memcached via
> unix domain socket or perhaps UDP, or any other ideas on how to fix
> this problem?

Adding unix socket support to ngx_http_memcached_module should be easy,
but I'm not sure if memcached suppot unix sockets.

BTW why do you serve local images from memcached ?
It's better for both CPU and memory to serve them from local filesystem
using sendfile.

Maybe I/O is too high ?
Use memcached can avoid this .

2008-08-01



Chancey



发件人： Igor Sysoev
发送时间： 2008-08-01  13:49:26
收件人： nginx@sysoev.ru
抄送：
主题： Re: Memcached module -- unix domain socket support? (too many 
TIME_WAITs..)

On Thu, Jul 31, 2008 at 09:54:03PM -0700, Kon Wilms wrote:

> 
> Is there any way to connect nginx's memcached module to memcached via
> unix domain socket or perhaps UDP, or any other ideas on how to fix
> this problem?

Adding unix socket support to ngx_http_memcached_module should be easy,
but I'm not sure if memcached suppot unix sockets.

BTW why do you serve local images from memcached ?
It's better for both CPU and memory to serve them from local filesystem
using sendfile.


--
Igor Sysoev
http://sysoev.ru/en/

On Fri, Aug 01, 2008 at 02:27:55PM +0800, Chancey wrote:

> Maybe I/O is too high ? 
> Use memcached can avoid this .

Using local memcached means that all data can fit in memory, therefore
local filesystem data will be cached by OS VM and there will not be
high disk I/O.

On Thu, Jul 31, 2008 at 10:43 PM, Igor Sysoev <is@rambler-co.ru> wrote:
> Adding unix socket support to ngx_http_memcached_module should be easy,
> but I'm not sure if memcached suppot unix sockets.

It does. Use for the memcached config:
-s <file>     unix socket path to listen on (disables network support)

> BTW why do you serve local images from memcached ?
> It's better for both CPU and memory to serve them from local filesystem
> using sendfile.

The images are coming from webcameras as motion JPEG, so each image is
updated 5 times a second on average. The same image is updated in
memcached, and fed to a flash player. Memcached is the most efficient
way to serve these without writing a custom streaming backend. Writing
to disk is definitely not an option.

Cheers
Kon

Hi,
> Adding unix socket support to ngx_http_memcached_module should be easy,
> but I'm not sure if memcached suppot unix sockets.
>
>   

Supporting unix socket would be cool.  memcached supports it.   I would
also recommend adding UDP support at the same time.

Yes, i meant this like you.
Using memcached can avoid disk I/O. I guess that is why he used 
memcached on localhost.

sorry , my english very pool.



2008-08-01



Chancey



发件人： Igor Sysoev
发送时间： 2008-08-01  14:39:19
收件人： nginx@sysoev.ru
抄送：
主题： Re: Re: Memcached module -- unix domain socket support? (too many 
TIME_WAITs..)

On Fri, Aug 01, 2008 at 02:27:55PM +0800, Chancey wrote:

> Maybe I/O is too high ? 
> Use memcached can avoid this .

Using local memcached means that all data can fit in memory, therefore
local filesystem data will be cached by OS VM and there will not be
high disk I/O.

>  > Nginx works fine, however, the problem I have is that thousands of
> 
> BTW why do you serve local images from memcached ?
> It's better for both CPU and memory to serve them from local filesystem
> using sendfile.
> 
> 
> -- 
> Igor Sysoev
> http://sysoev.ru/en/

--
Igor Sysoev
http://sysoev.ru/en/

nginx is great ! but it has only one mode and one request scheduling in 
Load balance.
if it has more options in load balance. it will be perfect

2008-08-01



Chancey



发件人： Igor Sysoev
发送时间： 2008-08-01  13:49:26
收件人： nginx@sysoev.ru
抄送：
主题： Re: Memcached module -- unix domain socket support? (too many 
TIME_WAITs..)

On Thu, Jul 31, 2008 at 09:54:03PM -0700, Kon Wilms wrote:

> 
> Is there any way to connect nginx's memcached module to memcached via
> unix domain socket or perhaps UDP, or any other ideas on how to fix
> this problem?

Adding unix socket support to ngx_http_memcached_module should be easy,
but I'm not sure if memcached suppot unix sockets.

BTW why do you serve local images from memcached ?
It's better for both CPU and memory to serve them from local filesystem
using sendfile.


--
Igor Sysoev
http://sysoev.ru/en/

Request scheduling as in queuing with priority, QoS?

Meaning, one or more location should have a higher priority in
completing the request, by going into the front of the queue?

For example, you have 1000 requests per second, and 10 of them are for
the location /realtime.jpg, and you want those requests to jump in front
of the queue and return the responses ASAP?

+1 point for such QOS

2008/8/1 Tit Petric <black@scene-si.org>

I wish nginx can't only have round-robin scheduling , it would be added 
more scheduling like LVS(IPVS).



2008-08-01



Chancey



发件人： Tit Petric
发送时间： 2008-08-01  15:56:40
收件人： nginx@sysoev.ru
抄送：
主题： Re: Direct Routing of load balancing would be added in feature?

Request scheduling as in queuing with priority, QoS?

Meaning, one or more location should have a higher priority in
completing the request, by going into the front of the queue?

For example, you have 1000 requests per second, and 10 of them are for
the location /realtime.jpg, and you want those requests to jump in front
of the queue and return the responses ASAP?

On Fri, Aug 01, 2008 at 04:36:02PM +0800, Chancey wrote:
> I wish nginx can't only have round-robin scheduling , it would be added more scheduling like LVS(IPVS).

See http://wiki.codemongers.com/NginxModules for the upstream_hash and
upstream_fair (weighted least-connection round-robin) load balancer 
modules.

Best regards,
 Grzegorz Nosek

Is anyone interested in implementing this feature for a bounty? :-) 
Igor?

Cheers
Kon

Kon Wilms wrote:
> I have a pool of memcached servers running on localhost with
> connection to nginx, feeding motion jpeg data out via static JPEGs to
> a flash application. The load is pretty high on the server because of
> this with each client running 3 GETs from memcached every second (i.e.
> 3fps).
> Cheers
> Kon

How do you think about use ncache module instead of memcached ? I'm 
definitly not an expert but when I read your post, it's the idea that I 
have had.