I'm trying to diagnose why we're using so much swap memory for Riak+Search, and derive what settings should be changed such that we can lower our memory utilization. Obviously we can scale up our riak boxes, but I'd like to have better insight into the settings before we just upgrade the machines.
We have a 512 partition 5 node ring with LevelDb Backend, Search enabled.
Currently we're seeing 100% of all available physical ram consumed by riak (1.7gb) and a large swap use as well: ~3.6gb:
riak 2677 1.7 85.1 3665736 1745696 pts/0 Ssl+ Mar12 109:16 /usr/lib/riak/erts-5.8.4/bin/beam.smp -K true -A 64 -W w --
My my computations according to http://wiki.basho.com/LevelDB.html we should be at around ~850mb of ram for the levelDb cache. I calculate another ~128 for JS VMs, but I have no way to figure out where the rest is used.
How do we figure out where the rest of the memory is being consumed, and how can we tune it to use less memory?
I could confirm this, my 2-node cluster have leveledb backend and high write intensity. After about half of hour it consume all RAM and begins slow down. I use etop to diagnose which erlang process holds the memory and didnt findt it. Thought htop says that all memory splitted between beam processes. Triyed differnet leveldb tweaks, doesnt change this behaviour. Riak was installed from official x64 deb package and got 1.1.1 version. Also I test this on my fedora box, and this is reproducible.
Btw, riak conrol web gui also says that my riak consume about 2% of memory, while all memory is holded by beam.