Localmemcache-0.2.0: The beauty of memcached. For local data. Blazingly fast

Ruby
1

Hi,

I am happy to announce the next version of localmemcache (0.2.0), a
library that aims to provide an interface similar to memcached but for
accessing local data instead of remote data. It’s based on mmap()'ed
shared memory for maximum speed.

Changes for 0.2.0:

  • Localmemcache uses logging now: In case your application is
    terminated
    while accessing the shared memory (eg by a kill -9), localmemcache
    is
    now able to restore the integrity of your data.
  • \0 character can now be used in keys and values
  • The ruby binding now features a keys() method.
  • Added a C API.

0.2.0 facts:

  • lightweight: The core library is just about 1300 lines of C code
  • fast: About 40% slower than Ruby’s hash (or about 36 times
    faster than a local memcached).

EXAMPLE

require ‘localmemcache’
$lm = LocalMemCache.new :namespace => “viewcounters”
$lm[:foo] = 1
$lm[:foo]
$lm.delete(:foo)

INSTALL

gem install localmemcache

(In case rubyforge has not yet updated the mirrors, fetch the 0.2.0 gem
from here: http://github.com/sck/localmemcache/downloads and then do

gem install localmemcache-0.2.0.gem )

CONTACT

Please contact me with bugs, suggestions and patches at: schween + snafu

de

LINKS

Localmemcache: http://localmemcache.rubyforge.org/
Rubyforge project: http://localmemcache.rubyforge.org/

Source code is hosted on github: GitHub - sck/localmemcache: mmap -> lmc_valloc -> hashtable. BAM database

Best,

Sven C. Koehler

afa5fe7a3a23930a50d0b8b4a1a55880 localmemcache-0.2.0.gem
050cab44c9a5b9cb6214779398c0cfb7 localmemcache-0.2.0.tar.gz

2

On Mar 30, 2009, at 5:17 PM, Sven C. Koehler wrote:

gem install localmemcache

really nice idea. can’t wait to try it out!

a @ http://codeforpeople.com/

3

From: “Sven C. Koehler” [email protected]

I am happy to announce the next version of localmemcache (0.2.0), a
library that aims to provide an interface similar to memcached but for
accessing local data instead of remote data. It’s based on mmap()'ed
shared memory for maximum speed.

Very nice!!

I’ve been daydreaming about a mmap’d hash table for a
couple years now – and I wasn’t even contemplating
having it able to be shared between processes, as it
appears you’ve done.

Your code appears nicely decoupled. It seems possible
if one were to write Windoze versions of lmc_lock.c and
lmc_shm.c , the rest of the code might just work on that
platform. (I may attempt this… someday…)

Anyway, thanks for releasing this interesting library!

Regards,

Bill

4

On Tue, Mar 31, 2009 at 04:13:39PM +0900, Bill K. wrote:

From: “Sven C. Koehler” [email protected]

I am happy to announce the next version of localmemcache (0.2.0), a
library that aims to provide an interface similar to memcached but for
accessing local data instead of remote data. It’s based on mmap()'ed
shared memory for maximum speed.

Very nice!!

Thanks!

Your code appears nicely decoupled. It seems possible
if one were to write Windoze versions of lmc_lock.c and
lmc_shm.c , the rest of the code might just work on that
platform. (I may attempt this… someday…)

Yeah, this should work. :wink:

-S.

5

On Tue, 31 Mar 2009 08:17:45 +0900
“Sven C. Koehler” [email protected] wrote:

Hi,

I am happy to announce the next version of localmemcache (0.2.0), a
library that aims to provide an interface similar to memcached but for
accessing local data instead of remote data. It’s based on mmap()'ed
shared memory for maximum speed.

This sounds great, can’t wait to add it as a caching option to Ramaze.

0.2.0 facts:
$lm[:foo]
$lm.delete(:foo)

I tried that, but $lm[:foo] always returned nil.
That’s on ArchLinux, both 32 and 64bit.

6

On Sat, Apr 4, 2009 at 6:36 PM, Michael F.
[email protected] wrote:

require ‘localmemcache’
$lm = LocalMemCache.new :namespace => “viewcounters”
$lm[:foo] = 1
$lm[:foo]
$lm.delete(:foo)

I tried that, but $lm[:foo] always returned nil.
That’s on ArchLinux, both 32 and 64bit.

The same for me. My configuration: ubuntu 8.10 64bit, ruby 1.8.7.

–
Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog

7

On Sat, Apr 04, 2009 at 07:51:07PM +0200, Sven C. Koehler wrote:

Hmm… Can you see the namespace in /var/tmp/localmemcache/ ?
Can you see the key in $lm.keys()?
Does ``du -h /var/tmp/localmemcache/viewcounters.lmc’’ show that there’s
content in the file?

This is how it looks on my machine:

| sck@comp ~/tmp $ uname -a
| Linux comp 2.6.23-gentoo-r3-b #6 SMP Sat Aug 23 08:57:32 UTC 2008
| x86_64 Intel® Xeon® CPU E5205 @ 1.86GHz GenuineIntel GNU/Linux
| sck@comp ~/tmp $ irb
| irb(main):001:0> require ‘localmemcache’
| => true
| irb(main):002:0> $lm = LocalMemCache.new :namespace => “viewcounters”
| => #LocalMemCache:0x2b5ed7ef0130
| irb(main):003:0> $lm.keys
| => []
| irb(main):004:0> $lm[:foo]
| => nil
| irb(main):005:0> $lm[:foo] = 1
| => 1
| irb(main):006:0> $lm[:foo]
| => “1”
| irb(main):007:0> $lm.keys
| => [“foo”]
| irb(main):008:0> sck@comp ~/tmp $
| sck@comp ~/tmp $ ls -al /var/tmp/localmemcache/viewcounters.lmc
| -rwxr-xr-x 1 sck sck 1073741824 Apr 4 17:52
| /var/tmp/localmemcache/viewcounters.lmc
| sck@comp ~/tmp $ du -h /var/tmp/localmemcache/viewcounters.lmc
| 16K /var/tmp/localmemcache/viewcounters.lmc
| sck@comp ~/tmp $ irb
| irb(main):002:0> require ‘localmemcache’
| => true
| irb(main):003:0> $lm = LocalMemCache.new :namespace => “viewcounters”
| => #LocalMemCache:0x2b50be3e6978
| irb(main):004:0> $lm.keys
| => [“foo”]
| irb(main):005:0>

-S.

8

On Sat, Apr 4, 2009 at 7:50 PM, Sven C. Koehler [email protected]
wrote:

Hmm… Can you see the namespace in /var/tmp/localmemcache/ ?

Yes
$ ls /var/tmp/localmemcache/
foo.lmc viewcounters.lmc

Can you see the key in $lm.keys()?
Yes, but they are duplicated after setting on the same key:

$lm = LocalMemCache.new :namespace => “viewcounters”
=> #LocalMemCache:0x7f9c05d844d0
$lm.keys
=> []
$lm[:foo] = 1
=> 1
$lm[:foo]
=> nil
$lm.keys
=> [“foo”]
$lm[:foo] = 2
=> 2
$lm[:foo]
=> nil
$lm.keys
=> [“foo”, “foo”]

Does ``du -h /var/tmp/localmemcache/viewcounters.lmc’’ show that there’s
content in the file?

$ du -h /var/tmp/localmemcache/viewcounters.lmc
20K /var/tmp/localmemcache/viewcounters.lmc

–
Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog

9

On Sun, Apr 05, 2009 at 01:36:09AM +0900, Michael F. wrote:

On Tue, 31 Mar 2009 08:17:45 +0900
“Sven C. Koehler” [email protected] wrote:

require ‘localmemcache’
$lm = LocalMemCache.new :namespace => “viewcounters”
$lm[:foo] = 1
$lm[:foo]
$lm.delete(:foo)

I tried that, but $lm[:foo] always returned nil.
That’s on ArchLinux, both 32 and 64bit.

Hmm… Can you see the namespace in /var/tmp/localmemcache/ ?
Can you see the key in $lm.keys()?
Does ``du -h /var/tmp/localmemcache/viewcounters.lmc’’ show that there’s
content in the file?

Best,

Sven

10

Ok, thanks to everyone who reported bugs!
I think I fixed this problem, so could anyone who had problems setting
keys run the following command to verify whether it works better now?

% git clone git://github.com/sck/localmemcache.git &&
cd localmemcache && rake sanity_test

The output at the end should look like this:

| LocalMemCache
| - should allow to set and query keys
| - should support the [] and []= operators
| - should allow deletion of keys
| - should return a list of keys
| - should support \0 in values and keys
| - should throw exception if pool is full
| - should support checking of namespaces
| - should support clearing of namespaces
|
| 8 specifications (8 requirements), 0 failures, 0 errors

(I haven’t fixed the close bug yet.)

Best,

Sven

11

On Sun, Apr 5, 2009 at 3:09 PM, Sven C. Koehler [email protected]
wrote:

| - should allow to set and query keys
| - should support the [] and []= operators
| - should allow deletion of keys
| - should return a list of keys
| - should support \0 in values and keys
| - should throw exception if pool is full
| - should support checking of namespaces
| - should support clearing of namespaces
|
| 8 specifications (8 requirements), 0 failures, 0 errors

It works as you expected:

LocalMemCache

  • should allow to set and query keys
  • should support the [] and []= operators
  • should allow deletion of keys
  • should return a list of keys
  • should support \0 in values and keys
  • should throw exception if pool is full
  • should support checking of namespaces
  • should support clearing of namespaces

8 specifications (8 requirements), 0 failures, 0 errors

Pozdrawiam

Rados³aw Bu³at
http://radarek.jogger.pl - mój blog

12

On Sat, 04 Apr 2009 12:59:56 -0500, Sven C. Koehler wrote:

This is how it looks on my machine:

I can reproduce the issue Michael and Radosław are having:

reima@marvin:~$ uname -a
Linux marvin 2.6.27-11-generic #1 SMP Thu Jan 29 19:24:39 UTC 2009
i686
GNU/Linux
reima@marvin:~$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]
reima@marvin:~$ irb

require ‘localmemcache’
=> true

$lm = LocalMemCache.new :namespace => “viewcounters”
=> #LocalMemCache:0xb7b69bdc

$lm.keys
=> []

$lm[:foo]
=> nil

$lm[:foo] = 1
=> 1

$lm[:foo]
=> nil

$lm.keys
=> [“foo”]

puts du -h /var/tmp/localmemcache/viewcounters.lmc
16K /var/tmp/localmemcache/viewcounters.lmc
=> nil

And on a sidenote:

reima@marvin:~$ irb

require ‘localmemcache’
=> true

$lm = LocalMemCache.new :namespace => “viewcounters”
=> #LocalMemCache:0xb79eac20

$lm.close
=> nil

$lm.keys
(irb):4: [BUG] Segmentation fault
ruby 1.8.7 (2008-08-11 patchlevel 72) [i486-linux]

Aborted
reima@marvin:~$

-Matthias

13

On Sun, 5 Apr 2009 22:09:04 +0900
“Sven C. Koehler” [email protected] wrote:

| - should allow to set and query keys
| - should support the [] and []= operators
| - should allow deletion of keys
| - should return a list of keys
| - should support \0 in values and keys
| - should throw exception if pool is full
| - should support checking of namespaces
| - should support clearing of namespaces
|
| 8 specifications (8 requirements), 0 failures, 0 errors

Aye, now it works as expected, thanks a lot. Will we get an updated
release as
well?