Riak/LevelDB Memory Usage

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak/LevelDB Memory Usage

Shane McEwan-2
G'day!

Our Riak nodes have 48GB of RAM in them. When we installed Riak 1.2.0 on
them we tuned the LevelDB settings as per the Parameter Planning section
in http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/

  {eleveldb, [
              {write_buffer_size_min, 31457280},
              {write_buffer_size_max, 62914560},
              {cache_size, 300000000},
              {max_open_files, 300}
             ]},

With 64 vnodes per node, an average SST size of 2MB, key size of 150B
and average value size of 1024B we were expecting our Riak memory usage
to peak at around 24GB . . . the recommended 50% of system memory.

Currently the Riak process on each of our nodes is using 32GB of RAM and
still rising.

I've noticed that since we moved to Riak 1.3.1 new SST files are no
longer 2MB but are now 100MB. Recalculating memory usage based on that
gives me an expected usage of 32GB. So maybe that explains the higher
memory usage than expected.

So, 2 questions:

Can we expect the memory usage to plateau soon?

The cache_size of 300MB should allow for one of our nodes to go down and
we'll still be under the 50% threshold when the 64 down vnodes are
shared amongst the remaining three nodes . . . except we're already
using more than the 24GB threshold (or the new 32GB threshold with 100MB
SSTs) with all four nodes still up. How can this be?

I'd appreciate if someone could do the math for me and come up with
suggested tuning parameters because I'm losing faith in the documentation:

Riak: 1.3.1
Nodes: 4
Total RAM per node: 48GB
Desired RAM allocated to Riak: 24GB
ring_creation_size: 256
Partitions per node: 64

Thanks!

Shane.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak/LevelDB Memory Usage

Matthew Von-Maszewski
Shane,

Several points for your consideration:

- "cache_size" is broken in 1.3.x, fixed in 1.4.x.  Any cache_size above the default 8M can ruin performance because some idiot (me) put a linear walk in the cache code.  Reverted for 1.4 and cache_size tested to 2G.

- do you have AAE (active anti-entropy) running?  AAE creates a parallel set of vnodes/databases to track hash trees of the primary vnodes.  This will add to your memory load.

- max_open_files in 1.3 puts a limit on the count of open files, but NOT on the amount of memory that is used by each file.  The larger .sst files create larger bloom filters which take up a greater amount of memory for same max_open_files setting.  Poor accounting.  1.4.x has a correction that changes file cache accounting from file count to amount of memory used per file (limit is 4Mbyte * max_open_files).  I am aware of one other 1.3.x site that had to reduce their max_open_files from 300 or more, to around 70 to stabilize memory.

That said, I have attached a spreadsheet that simplifies the math discussed on our web site.  I have plugged in your numbers and see a "balance" around cache_size=8388603 and max_open_files=200.  


I am currently writing a dynamic cache size model for Riak 2.0.  All the smoke and mirrors for memory sizing will then go away.  The code will automatically change caches as vnodes migrate to and from a given node (no need to "reserve 50%") … you just say total memory you want to allocate to Riak and walk away.

Matthew




On Sep 6, 2013, at 12:10 PM, Shane McEwan <[hidden email]> wrote:

> G'day!
>
> Our Riak nodes have 48GB of RAM in them. When we installed Riak 1.2.0 on them we tuned the LevelDB settings as per the Parameter Planning section in http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/
>
> {eleveldb, [
>             {write_buffer_size_min, 31457280},
>             {write_buffer_size_max, 62914560},
>             {cache_size, 300000000},
>             {max_open_files, 300}
>            ]},
>
> With 64 vnodes per node, an average SST size of 2MB, key size of 150B and average value size of 1024B we were expecting our Riak memory usage to peak at around 24GB . . . the recommended 50% of system memory.
>
> Currently the Riak process on each of our nodes is using 32GB of RAM and still rising.
>
> I've noticed that since we moved to Riak 1.3.1 new SST files are no longer 2MB but are now 100MB. Recalculating memory usage based on that gives me an expected usage of 32GB. So maybe that explains the higher memory usage than expected.
>
> So, 2 questions:
>
> Can we expect the memory usage to plateau soon?
>
> The cache_size of 300MB should allow for one of our nodes to go down and we'll still be under the 50% threshold when the 64 down vnodes are shared amongst the remaining three nodes . . . except we're already using more than the 24GB threshold (or the new 32GB threshold with 100MB SSTs) with all four nodes still up. How can this be?
>
> I'd appreciate if someone could do the math for me and come up with suggested tuning parameters because I'm losing faith in the documentation:
>
> Riak: 1.3.1
> Nodes: 4
> Total RAM per node: 48GB
> Desired RAM allocated to Riak: 24GB
> ring_creation_size: 256
> Partitions per node: 64
>
> Thanks!
>
> Shane.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

LevelDB1.2MemModel.Shane.xls (40K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Riak/LevelDB Memory Usage

Shane McEwan-2
Thanks Matthew.

We've reduced the cache_size to 8388603 and max_open_files to 200 so
we'll see how that goes. We'll know in a month or two whether the memory
usage levels out where we expect. We are running AAE so I'm sure that
was having an effect on memory as well.

You cache changes in Riak 2.0 sound really good. Much easier to control
memory usage.

Thanks again!

On 06/09/13 18:46, Matthew Von-Maszewski wrote:

> Shane,
>
> Several points for your consideration:
>
> - "cache_size" is broken in 1.3.x, fixed in 1.4.x.  Any cache_size above the default 8M can ruin performance because some idiot (me) put a linear walk in the cache code.  Reverted for 1.4 and cache_size tested to 2G.
>
> - do you have AAE (active anti-entropy) running?  AAE creates a parallel set of vnodes/databases to track hash trees of the primary vnodes.  This will add to your memory load.
>
> - max_open_files in 1.3 puts a limit on the count of open files, but NOT on the amount of memory that is used by each file.  The larger .sst files create larger bloom filters which take up a greater amount of memory for same max_open_files setting.  Poor accounting.  1.4.x has a correction that changes file cache accounting from file count to amount of memory used per file (limit is 4Mbyte * max_open_files).  I am aware of one other 1.3.x site that had to reduce their max_open_files from 300 or more, to around 70 to stabilize memory.
>
> That said, I have attached a spreadsheet that simplifies the math discussed on our web site.  I have plugged in your numbers and see a "balance" around cache_size=8388603 and max_open_files=200.
>
>
> I am currently writing a dynamic cache size model for Riak 2.0.  All the smoke and mirrors for memory sizing will then go away.  The code will automatically change caches as vnodes migrate to and from a given node (no need to "reserve 50%") … you just say total memory you want to allocate to Riak and walk away.
>
> Matthew
>
>
>
>
>
> On Sep 6, 2013, at 12:10 PM, Shane McEwan <[hidden email]> wrote:
>
>> G'day!
>>
>> Our Riak nodes have 48GB of RAM in them. When we installed Riak 1.2.0 on them we tuned the LevelDB settings as per the Parameter Planning section in http://docs.basho.com/riak/1.2.0/tutorials/choosing-a-backend/LevelDB/
>>
>> {eleveldb, [
>>              {write_buffer_size_min, 31457280},
>>              {write_buffer_size_max, 62914560},
>>              {cache_size, 300000000},
>>              {max_open_files, 300}
>>             ]},
>>
>> With 64 vnodes per node, an average SST size of 2MB, key size of 150B and average value size of 1024B we were expecting our Riak memory usage to peak at around 24GB . . . the recommended 50% of system memory.
>>
>> Currently the Riak process on each of our nodes is using 32GB of RAM and still rising.
>>
>> I've noticed that since we moved to Riak 1.3.1 new SST files are no longer 2MB but are now 100MB. Recalculating memory usage based on that gives me an expected usage of 32GB. So maybe that explains the higher memory usage than expected.
>>
>> So, 2 questions:
>>
>> Can we expect the memory usage to plateau soon?
>>
>> The cache_size of 300MB should allow for one of our nodes to go down and we'll still be under the 50% threshold when the 64 down vnodes are shared amongst the remaining three nodes . . . except we're already using more than the 24GB threshold (or the new 32GB threshold with 100MB SSTs) with all four nodes still up. How can this be?
>>
>> I'd appreciate if someone could do the math for me and come up with suggested tuning parameters because I'm losing faith in the documentation:
>>
>> Riak: 1.3.1
>> Nodes: 4
>> Total RAM per node: 48GB
>> Desired RAM allocated to Riak: 24GB
>> ring_creation_size: 256
>> Partitions per node: 64
>>
>> Thanks!
>>
>> Shane.
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com