Crash Log: yz_anti_entropy

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Crash Log: yz_anti_entropy

Junk, Damion A
Hi All -

I'm seeing some strange behavior on a dev server. Right now the riak kv 2.2.0 server has 9GB free Ram, and no keys/values stored in Riak at all. (I've been trying to figure out what's going on with the server, so I completely wiped /var/lib/riak and re-installed from packagecould). Ulimit -n is set appropriately as well.

If I make the following changes to /etc/riak/riak.conf I get crash error messages:

storage_backend = leveldb
search = on

Turning search on with bitcask results in no such crash/error messages. When switching between bitcask/leveldb on this test instance, i'm wiping the /var/lib/riak directory each time.

Not much is turning up in google, and there is definitely plenty of memory available for allocation.


Just wondering if anyone else has seen this.


Thanks!


The error messages:

(crash.log)

2017-01-18 15:27:23 =CRASH REPORT====
  crasher:
    initial call: yz_index_hashtree:init/1
    pid: <0.2356.0>
    registered_name: []
    exception exit: {{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
    ancestors: [yz_index_hashtree_sup,yz_general_sup,yz_sup,<0.711.0>]
    messages: []
    links: [<0.2206.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 689
  neighbours:
2017-01-18 15:27:24 =SUPERVISOR REPORT====
     Supervisor: {local,yz_index_hashtree_sup}
     Context:    child_terminated
     Reason:     {{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
     Offender:   [{pid,<0.2356.0>},{name,ignored},{mfargs,{yz_index_hashtree,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]



(console.log)

2017-01-18 15:35:39.100 [error] <0.3049.0> CRASH REPORT Process <0.3049.0> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in gen_server:init_it/6 line 328
2017-01-18 15:35:39.101 [error] <0.2206.0> Supervisor yz_index_hashtree_sup had child ignored started with {yz_index_hashtree,start_link,undefined} at <0.3049.0> exit with reason no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in context child_terminated
2017-01-18 15:35:39.101 [info] <0.3050.0>@riak_core_throttle:enable_throttle:119 Enabling throttle for yokozuna/aae_throttle.
2017-01-18 15:35:39.101 [error] <0.3030.0> gen_server yz_entropy_mgr terminated with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371
2017-01-18 15:35:39.101 [error] <0.3030.0> CRASH REPORT Process yz_entropy_mgr with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in gen_server:terminate/6 line 744
2017-01-18 15:35:39.102 [error] <0.2010.0> Supervisor yz_general_sup had child yz_entropy_mgr started with yz_entropy_mgr:start_link() at <0.3030.0> exit with reason no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in context child_terminated
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Crash Log: yz_anti_entropy

Magnus Kessler
On 18 January 2017 at 20:36, Junk, Damion A <[hidden email]> wrote:
Hi All -

I'm seeing some strange behavior on a dev server. Right now the riak kv 2.2.0 server has 9GB free Ram, and no keys/values stored in Riak at all. (I've been trying to figure out what's going on with the server, so I completely wiped /var/lib/riak and re-installed from packagecould). Ulimit -n is set appropriately as well.

If I make the following changes to /etc/riak/riak.conf I get crash error messages:

storage_backend = leveldb
search = on

Turning search on with bitcask results in no such crash/error messages. When switching between bitcask/leveldb on this test instance, i'm wiping the /var/lib/riak directory each time.

Not much is turning up in google, and there is definitely plenty of memory available for allocation.


Just wondering if anyone else has seen this.


Thanks!


The error messages:

(crash.log)

2017-01-18 15:27:23 =CRASH REPORT====
  crasher:
    initial call: yz_index_hashtree:init/1
    pid: <0.2356.0>
    registered_name: []
    exception exit: {{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
    ancestors: [yz_index_hashtree_sup,yz_general_sup,yz_sup,<0.711.0>]
    messages: []
    links: [<0.2206.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 27
    reductions: 689
  neighbours:
2017-01-18 15:27:24 =SUPERVISOR REPORT====
     Supervisor: {local,yz_index_hashtree_sup}
     Context:    child_terminated
     Reason:     {{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},{yz_index_hashtree,init,1,[{file,"src/yz_index_hashtree.erl"},{line,191}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
     Offender:   [{pid,<0.2356.0>},{name,ignored},{mfargs,{yz_index_hashtree,start_link,undefined}},{restart_type,temporary},{shutdown,5000},{child_type,worker}]



(console.log)

2017-01-18 15:35:39.100 [error] <0.3049.0> CRASH REPORT Process <0.3049.0> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in gen_server:init_it/6 line 328
2017-01-18 15:35:39.101 [error] <0.2206.0> Supervisor yz_index_hashtree_sup had child ignored started with {yz_index_hashtree,start_link,undefined} at <0.3049.0> exit with reason no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}} in hashtree:new_segment_store/2 line 725 in context child_terminated
2017-01-18 15:35:39.101 [info] <0.3050.0>@riak_core_throttle:enable_throttle:119 Enabling throttle for yokozuna/aae_throttle.
2017-01-18 15:35:39.101 [error] <0.3030.0> gen_server yz_entropy_mgr terminated with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371
2017-01-18 15:35:39.101 [error] <0.3030.0> CRASH REPORT Process yz_entropy_mgr with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in gen_server:terminate/6 line 744
2017-01-18 15:35:39.102 [error] <0.2010.0> Supervisor yz_general_sup had child yz_entropy_mgr started with yz_entropy_mgr:start_link() at <0.3030.0> exit with reason no match of right hand value {error,{{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,725}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{yz_index_hashtree,do_new_tree,3,[{file,"src/yz_index_hashtree.erl"},{line,377}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{yz_index_hashtree,init_trees,3,[{file,"src/yz_index_hashtree.erl"},{line,340}]},...]}} in yz_entropy_mgr:'-reload_hashtrees/3-fun-0-'/2 line 371 in context child_terminated

Hi Damion,

Let me first state that AAE always uses leveldb, regardless of the storage backend chosen for Riak KV data. Could you please state how much physical memory your Riak nodes have, and what you have configured for "leveldb.maximum_memory.percent" in "riak.conf"? Have you changed the settings for "search.solr.jvm_options", in particular the memory allocated to Solr?

As a general rule, leveldb should have at least 350MB of memory available per partition, and performance has been shown to increase with up to 2GB (2.5 GB when also using Search and AAE) per partition. Please check that you have enough memory available in your system.

Kind Regards,

Magnus
 
--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Crash Log: yz_anti_entropy

Junk, Damion A
Hi Magnus -

I've tried a wide range of parameters for leveldb.maximum_memory_percent ranging from 5 to 70. I also tried the leveldb.maximum_memory setting in bytes, ranging from 500MB to 4GB. I get the same results in the crash/console log no matter what the settings. But the log messages seem to indicate an issue with yokozuna, and not leveldb itself from what I can tell.

I set the max (-Xmx) to 2G for SOLR as well.

From the log messages, it looks like it's not actually the KV leveldb system that's crashing, but the yokozuna system. I'm not sure how to control or set memory here:

{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}

This is a development node, running as a single (nojn-clustered) riak node. It has 14G memory, and at the time of trying changes with Riak, 9GB were free. 


To Recap:

There are no keys/values in the database at all. 
The only default settings I changed were:

storage_backend = leveldb
search = on

and when that didn't work, I started changing:

search.solr.jvm_options = -d64 -Xms1g -Xmx2g -XX:+UseStringCache -XX:+UseCompressedOops
leveldb.maximum_memory_percent = 5 .. 70 

and then when nothing seemed to change:

leveldb.maximum_memory =  1000000 ... 4000000000


Thanks for any assistance!


Damion


On Jan 19, 2017, at 3:33 AM, Magnus Kessler <[hidden email]> wrote:

Hi Damion,

Let me first state that AAE always uses leveldb, regardless of the storage backend chosen for Riak KV data. Could you please state how much physical memory your Riak nodes have, and what you have configured for "leveldb.maximum_memory.percent" in "riak.conf"? Have you changed the settings for "search.solr.jvm_options", in particular the memory allocated to Solr?

As a general rule, leveldb should have at least 350MB of memory available per partition, and performance has been shown to increase with up to 2GB (2.5 GB when also using Search and AAE) per partition. Please check that you have enough memory available in your system.

Kind Regards,

Magnus
 
-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Crash Log: yz_anti_entropy

Matthew Von-Maszewski
Damion,

Add the following settings within riak.conf:

leveldb.limited_developer_mem = on
ring_size = 8

Erase all data / vnodes and start over.

Matthew


On Jan 19, 2017, at 8:51 AM, Junk, Damion A <[hidden email]> wrote:

Hi Magnus -

I've tried a wide range of parameters for leveldb.maximum_memory_percent ranging from 5 to 70. I also tried the leveldb.maximum_memory setting in bytes, ranging from 500MB to 4GB. I get the same results in the crash/console log no matter what the settings. But the log messages seem to indicate an issue with yokozuna, and not leveldb itself from what I can tell.

I set the max (-Xmx) to 2G for SOLR as well.

From the log messages, it looks like it's not actually the KV leveldb system that's crashing, but the yokozuna system. I'm not sure how to control or set memory here:

{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}

This is a development node, running as a single (nojn-clustered) riak node. It has 14G memory, and at the time of trying changes with Riak, 9GB were free. 


To Recap:

There are no keys/values in the database at all. 
The only default settings I changed were:

storage_backend = leveldb
search = on

and when that didn't work, I started changing:

search.solr.jvm_options = -d64 -Xms1g -Xmx2g -XX:+UseStringCache -XX:+UseCompressedOops
leveldb.maximum_memory_percent = 5 .. 70 

and then when nothing seemed to change:

leveldb.maximum_memory =  1000000 ... 4000000000


Thanks for any assistance!


Damion


On Jan 19, 2017, at 3:33 AM, Magnus Kessler <[hidden email]> wrote:

Hi Damion,

Let me first state that AAE always uses leveldb, regardless of the storage backend chosen for Riak KV data. Could you please state how much physical memory your Riak nodes have, and what you have configured for "leveldb.maximum_memory.percent" in "riak.conf"? Have you changed the settings for "search.solr.jvm_options", in particular the memory allocated to Solr?

As a general rule, leveldb should have at least 350MB of memory available per partition, and performance has been shown to increase with up to 2GB (2.5 GB when also using Search and AAE) per partition. Please check that you have enough memory available in your system.

Kind Regards,

Magnus
 
-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Crash Log: yz_anti_entropy

Junk, Damion A
Matthew -

That did it! 

Actually, I tried with both settings, and also with just the ring_size change. 

Setting ring_size to 8 got rid of crashing.  I'll have to do a bit more reading on this setting I suppose. I have a much more memory-constrained virtual machine running on my local desktop running with just the default install settings and no crashing. 

Thanks!

Damion

On Jan 19, 2017, at 7:57 AM, Matthew Von-Maszewski <[hidden email]> wrote:

Damion,

Add the following settings within riak.conf:

leveldb.limited_developer_mem = on
ring_size = 8

Erase all data / vnodes and start over.

Matthew


On Jan 19, 2017, at 8:51 AM, Junk, Damion A <[hidden email]> wrote:

Hi Magnus -

I've tried a wide range of parameters for leveldb.maximum_memory_percent ranging from 5 to 70. I also tried the leveldb.maximum_memory setting in bytes, ranging from 500MB to 4GB. I get the same results in the crash/console log no matter what the settings. But the log messages seem to indicate an issue with yokozuna, and not leveldb itself from what I can tell.

I set the max (-Xmx) to 2G for SOLR as well.

From the log messages, it looks like it's not actually the KV leveldb system that's crashing, but the yokozuna system. I'm not sure how to control or set memory here:

{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}

This is a development node, running as a single (nojn-clustered) riak node. It has 14G memory, and at the time of trying changes with Riak, 9GB were free. 


To Recap:

There are no keys/values in the database at all. 
The only default settings I changed were:

storage_backend = leveldb
search = on

and when that didn't work, I started changing:

search.solr.jvm_options = -d64 -Xms1g -Xmx2g -XX:+UseStringCache -XX:+UseCompressedOops
leveldb.maximum_memory_percent = 5 .. 70 

and then when nothing seemed to change:

leveldb.maximum_memory =  1000000 ... 4000000000


Thanks for any assistance!


Damion


On Jan 19, 2017, at 3:33 AM, Magnus Kessler <[hidden email]> wrote:

Hi Damion,

Let me first state that AAE always uses leveldb, regardless of the storage backend chosen for Riak KV data. Could you please state how much physical memory your Riak nodes have, and what you have configured for "leveldb.maximum_memory.percent" in "riak.conf"? Have you changed the settings for "search.solr.jvm_options", in particular the memory allocated to Solr?

As a general rule, leveldb should have at least 350MB of memory available per partition, and performance has been shown to increase with up to 2GB (2.5 GB when also using Search and AAE) per partition. Please check that you have enough memory available in your system.

Kind Regards,

Magnus
 
-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Crash Log: yz_anti_entropy

Matthew Von-Maszewski
Damion,

I will explain what happened.  

ring_size = 8:  The default ring_size is 64.  It is based on the recommendation of five servers for a minimum cluster.  You stated you are using only one machine.  64 divided by 5 is 12.8 vnodes per server ... and ring size needs to be a power of 2.  So next smaller power of 2 from 12 is 8.

leveldb.limited_developer_mem = on:  leveldb allocates certain memory buffer size per vnode (ring_size).  This setting reduces the size of those buffers by 10x.  

The two settings squash the memory requirements to give you a better opportunity for happiness with both search and riak on a single server.

Matthew



On Jan 19, 2017, at 9:15 AM, Junk, Damion A <[hidden email]> wrote:

Matthew -

That did it! 

Actually, I tried with both settings, and also with just the ring_size change. 

Setting ring_size to 8 got rid of crashing.  I'll have to do a bit more reading on this setting I suppose. I have a much more memory-constrained virtual machine running on my local desktop running with just the default install settings and no crashing. 

Thanks!

Damion

On Jan 19, 2017, at 7:57 AM, Matthew Von-Maszewski <[hidden email]> wrote:

Damion,

Add the following settings within riak.conf:

leveldb.limited_developer_mem = on
ring_size = 8

Erase all data / vnodes and start over.

Matthew


On Jan 19, 2017, at 8:51 AM, Junk, Damion A <[hidden email]> wrote:

Hi Magnus -

I've tried a wide range of parameters for leveldb.maximum_memory_percent ranging from 5 to 70. I also tried the leveldb.maximum_memory setting in bytes, ranging from 500MB to 4GB. I get the same results in the crash/console log no matter what the settings. But the log messages seem to indicate an issue with yokozuna, and not leveldb itself from what I can tell.

I set the max (-Xmx) to 2G for SOLR as well.

From the log messages, it looks like it's not actually the KV leveldb system that's crashing, but the yokozuna system. I'm not sure how to control or set memory here:

{badmatch,{error,{db_open,"IO error: lock /var/lib/riak/yz_anti_entropy/639406966332270026714112114313373821099470487552/LOCK: Cannot allocate memory"}

This is a development node, running as a single (nojn-clustered) riak node. It has 14G memory, and at the time of trying changes with Riak, 9GB were free. 


To Recap:

There are no keys/values in the database at all. 
The only default settings I changed were:

storage_backend = leveldb
search = on

and when that didn't work, I started changing:

search.solr.jvm_options = -d64 -Xms1g -Xmx2g -XX:+UseStringCache -XX:+UseCompressedOops
leveldb.maximum_memory_percent = 5 .. 70 

and then when nothing seemed to change:

leveldb.maximum_memory =  1000000 ... 4000000000


Thanks for any assistance!


Damion


On Jan 19, 2017, at 3:33 AM, Magnus Kessler <[hidden email]> wrote:

Hi Damion,

Let me first state that AAE always uses leveldb, regardless of the storage backend chosen for Riak KV data. Could you please state how much physical memory your Riak nodes have, and what you have configured for "leveldb.maximum_memory.percent" in "riak.conf"? Have you changed the settings for "search.solr.jvm_options", in particular the memory allocated to Solr?

As a general rule, leveldb should have at least 350MB of memory available per partition, and performance has been shown to increase with up to 2GB (2.5 GB when also using Search and AAE) per partition. Please check that you have enough memory available in your system.

Kind Regards,

Magnus
 
-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com