Riak 2.1.4 crashes with Out of Memory Error

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Riak 2.1.4 crashes with Out of Memory Error

Arulappan, Jerald (Jerald)

Hi,

I am using a single node riak server 2.1.4 with bitcask as backend for storing files.
The riak node stops working after every week. (Looks like when the active anti-entropy process recreates the hash tree)
The sylog shows Out of memory Error. But the console.log shows "sst: No such file or directory"
Syslog Error:

Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:37 TLCCBAPRO2 kernel: Killed process 16987 (sh) total-vm:106168kB, anon-rss:116kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30374 (memsup) total-vm:4112kB, anon-rss:80kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 14351 (cpu_sup) total-vm:4112kB, anon-rss:68kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30385 (sh) total-vm:106164kB, anon-rss:136kB, file-rss:416kB
Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang closed the connection.

Console.log:

2017-04-26 17:37:03.493 [info] <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/91343852333181432387730302044767688728495783936: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process <0.30178.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328
2017-04-26 17:37:03.515 [info] <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/45671926166590716193865151022383844364247891968: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process <0.30207.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328

 

The complete logs are in the attached zip file. Any thoughts on the root cause and possible solution to overcome this is much appreciated.

 

Regards,

Jerald

 


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

riak.zip (2M) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak 2.1.4 crashes with Out of Memory Error

Luke Bakken
Hi Jerald -

What is the average size of an object that you are storing in Riak?

I'm also seeing a lot of errors in the logs related to AAE, like you
note. I know there have been fixes in that part of Riak since 2.1.4
and recommend upgrading.

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 2, 2017 at 6:56 AM, Arulappan, Jerald (Jerald)
<[hidden email]> wrote:

> Hi,
>
> I am using a single node riak server 2.1.4 with bitcask as backend for
> storing files.
> The riak node stops working after every week. (Looks like when the active
> anti-entropy process recreates the hash tree)
> The sylog shows Out of memory Error. But the console.log shows "sst: No such
> file or directory"
> Syslog Error:
>
> Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685
> (beam.smp) score 824 or sacrifice child
> Apr 26 17:39:37 TLCCBAPRO2 kernel: Killed process 16987 (sh)
> total-vm:106168kB, anon-rss:116kB, file-rss:0kB
> Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685
> (beam.smp) score 824 or sacrifice child
> Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30374 (memsup)
> total-vm:4112kB, anon-rss:80kB, file-rss:0kB
> Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685
> (beam.smp) score 824 or sacrifice child
> Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 14351 (cpu_sup)
> total-vm:4112kB, anon-rss:68kB, file-rss:0kB
> Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685
> (beam.smp) score 824 or sacrifice child
> Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30385 (sh)
> total-vm:106164kB, anon-rss:136kB, file-rss:416kB
> Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang closed the connection.
>
> Console.log:
>
> 2017-04-26 17:37:03.493 [info]
> <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227
> riak_kv/91343852333181432387730302044767688728495783936: unable to start
> index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
> ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst:
> No such file or
> directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
> 2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process
> <0.30178.2881> with 0 neighbours exited with reason: no match of right hand
> value {error,{db_open,"IO error:
> ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst:
> No such file or directory"}} in hashtree:new_segment_store/2 line 675 in
> gen_server:init_it/6 line 328
> 2017-04-26 17:37:03.515 [info]
> <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227
> riak_kv/45671926166590716193865151022383844364247891968: unable to start
> index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
> ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst:
> No such file or
> directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
> 2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process
> <0.30207.2881> with 0 neighbours exited with reason: no match of right hand
> value {error,{db_open,"IO error:
> ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst:
> No such file or directory"}} in hashtree:new_segment_store/2 line 675 in
> gen_server:init_it/6 line 328
>
>
>
> The complete logs are in the attached zip file. Any thoughts on the root
> cause and possible solution to overcome this is much appreciated.
>
>
>
> Regards,
>
> Jerald
>
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Riak 2.1.4 crashes with Out of Memory Error

Arulappan, Jerald (Jerald)
Hi Luke Bakken,

The size of the objects stored in Riak varies from 10kb to 500kb.
I did see the need to upgrade RIAK due to AAE fixes and will be doing that.
However I wanted to understand the root cause and make sure  this wouldn't  happen again after the upgrade.

Regards,
Jerald

-----Original Message-----
From: Luke Bakken [mailto:[hidden email]]
Sent: Wednesday, May 03, 2017 8:01 PM
To: Arulappan, Jerald (Jerald) **CTR** <[hidden email]>
Cc: [hidden email]
Subject: Re: Riak 2.1.4 crashes with Out of Memory Error

Hi Jerald -

What is the average size of an object that you are storing in Riak?

I'm also seeing a lot of errors in the logs related to AAE, like you note. I know there have been fixes in that part of Riak since 2.1.4 and recommend upgrading.

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 2, 2017 at 6:56 AM, Arulappan, Jerald (Jerald) <[hidden email]> wrote:

> Hi,
>
> I am using a single node riak server 2.1.4 with bitcask as backend for
> storing files.
> The riak node stops working after every week. (Looks like when the
> active anti-entropy process recreates the hash tree) The sylog shows
> Out of memory Error. But the console.log shows "sst: No such file or
> directory"
> Syslog Error:
>
> Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685
> (beam.smp) score 824 or sacrifice child Apr 26 17:39:37 TLCCBAPRO2
> kernel: Killed process 16987 (sh) total-vm:106168kB, anon-rss:116kB,
> file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill
> process 16685
> (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2
> kernel: Killed process 30374 (memsup) total-vm:4112kB, anon-rss:80kB,
> file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill
> process 16685
> (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2
> kernel: Killed process 14351 (cpu_sup) total-vm:4112kB, anon-rss:68kB,
> file-rss:0kB Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill
> process 16685
> (beam.smp) score 824 or sacrifice child Apr 26 17:39:41 TLCCBAPRO2
> kernel: Killed process 30385 (sh) total-vm:106164kB, anon-rss:136kB,
> file-rss:416kB Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang
> closed the connection.
>
> Console.log:
>
> 2017-04-26 17:37:03.493 [info]
> <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227
> riak_kv/91343852333181432387730302044767688728495783936: unable to
> start
> index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
> ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst:
> No such file or
> directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"
> },{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]}
> ,{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hasht
> ree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]
> },{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hasht
> ree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak
> _kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"ge
> n_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l
> ib.erl"},{line,239}]}]}}
> 2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process
> <0.30178.2881> with 0 neighbours exited with reason: no match of right
> hand value {error,{db_open,"IO error:
> ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst:
> No such file or directory"}} in hashtree:new_segment_store/2 line 675
> in
> gen_server:init_it/6 line 328
> 2017-04-26 17:37:03.515 [info]
> <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227
> riak_kv/45671926166590716193865151022383844364247891968: unable to
> start
> index_hashtree: {error,{{badmatch,{error,{db_open,"IO error:
> ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst:
> No such file or
> directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"
> },{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]}
> ,{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hasht
> ree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]
> },{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hasht
> ree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak
> _kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"ge
> n_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_l
> ib.erl"},{line,239}]}]}}
> 2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process
> <0.30207.2881> with 0 neighbours exited with reason: no match of right
> hand value {error,{db_open,"IO error:
> ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst:
> No such file or directory"}} in hashtree:new_segment_store/2 line 675
> in
> gen_server:init_it/6 line 328
>
>
>
> The complete logs are in the attached zip file. Any thoughts on the
> root cause and possible solution to overcome this is much appreciated.
>
>
>
> Regards,
>
> Jerald
>
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.basho.com_ma
> ilman_listinfo_riak-2Dusers-5Flists.basho.com&d=DwIFaQ&c=BFpWQw8bsuKpl
> 1SgiZH64Q&r=zOPuaocGAfHxJUtnOg9ZgYW14EJAnNldsbCwB84w4zE&m=7t76DY3mCrip
> DH3glx_DHTdBaJGT_aqr1EtXkWhPYhI&s=rVfw97EO99wQM0eCfFhWyIPzTLyS2jk_3qZw
> XcvWcpE&e=
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak 2.1.4 crashes with Out of Memory Error

Magnus Kessler
In reply to this post by Arulappan, Jerald (Jerald)
On 2 May 2017 at 14:56, Arulappan, Jerald (Jerald) <[hidden email]> wrote:

Hi,

I am using a single node riak server 2.1.4 with bitcask as backend for storing files.
The riak node stops working after every week. (Looks like when the active anti-entropy process recreates the hash tree)
The sylog shows Out of memory Error. But the console.log shows "sst: No such file or directory"
Syslog Error:

Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:37 TLCCBAPRO2 kernel: Killed process 16987 (sh) total-vm:106168kB, anon-rss:116kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30374 (memsup) total-vm:4112kB, anon-rss:80kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 14351 (cpu_sup) total-vm:4112kB, anon-rss:68kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30385 (sh) total-vm:106164kB, anon-rss:136kB, file-rss:416kB
Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang closed the connection.

Console.log:

2017-04-26 17:37:03.493 [info] <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/91343852333181432387730302044767688728495783936: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process <0.30178.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328
2017-04-26 17:37:03.515 [info] <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/45671926166590716193865151022383844364247891968: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process <0.30207.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328

 

The complete logs are in the attached zip file. Any thoughts on the root cause and possible solution to overcome this is much appreciated.

 

Regards,

Jerald

 

 

Hi Jerald,

I suspect that there is a mis-configuration on your setup. Please check by running `riak config effective | grep "_dir"`, what the values of `platform_data_dir` and `anti_entropy.data_dir` are set to.

Kind Regards,

Magnus


-- 
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Riak 2.1.4 crashes with Out of Memory Error

Arulappan, Jerald (Jerald)

Hi Magnus Kessler,

 

The configuration looks good.

 

[root@server205 bin]# ./riak config effective | grep "_dir"

anti_entropy.data_dir = $(platform_data_dir)/anti_entropy

bitcask.data_root = $(platform_data_dir)/bitcask

leveldb.data_root = $(platform_data_dir)/leveldb

log.console.file = $(platform_log_dir)/console.log

log.crash.file = $(platform_log_dir)/crash.log

log.error.file = $(platform_log_dir)/error.log

platform_bin_dir = ./bin

platform_data_dir = ./data

platform_etc_dir = ./etc

platform_lib_dir = ./lib

platform_log_dir = ./log

ring.state_dir = $(platform_data_dir)/ring

search.anti_entropy.data_dir = $(platform_data_dir)/yz_anti_entropy

search.root_dir = $(platform_data_dir)/yz

search.temp_dir = $(platform_data_dir)/yz_temp

 

Regards,

Jerald

Emerging Products and Technologies | Avaya 

Voice +91 44 6108 4258 | [hidden email] | Chennai, India

 

From: Magnus Kessler [mailto:[hidden email]]
Sent: Thursday, May 04, 2017 12:36 PM
To: Arulappan, Jerald (Jerald) **CTR** <[hidden email]>
Cc: [hidden email]
Subject: Re: Riak 2.1.4 crashes with Out of Memory Error

 

On 2 May 2017 at 14:56, Arulappan, Jerald (Jerald) <[hidden email]> wrote:

Hi,

I am using a single node riak server 2.1.4 with bitcask as backend for storing files.
The riak node stops working after every week. (Looks like when the active anti-entropy process recreates the hash tree)
The sylog shows Out of memory Error. But the console.log shows "sst: No such file or directory"
Syslog Error:

Apr 26 17:39:37 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:37 TLCCBAPRO2 kernel: Killed process 16987 (sh) total-vm:106168kB, anon-rss:116kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30374 (memsup) total-vm:4112kB, anon-rss:80kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 14351 (cpu_sup) total-vm:4112kB, anon-rss:68kB, file-rss:0kB
Apr 26 17:39:41 TLCCBAPRO2 kernel: Out of memory: Kill process 16685 (beam.smp) score 824 or sacrifice child
Apr 26 17:39:41 TLCCBAPRO2 kernel: Killed process 30385 (sh) total-vm:106164kB, anon-rss:136kB, file-rss:416kB
Apr 26 17:44:48 TLCCBAPRO2 run_erl[16682]: Erlang closed the connection.

Console.log:

2017-04-26 17:37:03.493 [info] <0.625.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/91343852333181432387730302044767688728495783936: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/91343852333181432387730302044767688728495783936/sst_0/001954.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-04-26 17:37:03.515 [error] <0.30178.2881> CRASH REPORT Process <0.30178.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/936274486415109681974235595958868809467081785344/000037.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328
2017-04-26 17:37:03.515 [info] <0.623.0>@riak_kv_vnode:maybe_create_hashtrees:227 riak_kv/45671926166590716193865151022383844364247891968: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_hashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}}
2017-04-26 17:37:03.516 [error] <0.30207.2881> CRASH REPORT Process <0.30207.2881> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: ./data/anti_entropy/45671926166590716193865151022383844364247891968/sst_0/002239.sst: No such file or directory"}} in hashtree:new_segment_store/2 line 675 in gen_server:init_it/6 line 328

 

The complete logs are in the attached zip file. Any thoughts on the root cause and possible solution to overcome this is much appreciated.

 

Regards,

Jerald

 

 

 

Hi Jerald,

 

I suspect that there is a mis-configuration on your setup. Please check by running `riak config effective | grep "_dir"`, what the values of `platform_data_dir` and `anti_entropy.data_dir` are set to.

 

Kind Regards,

 

Magnus

 

 

-- 

Magnus Kessler

Client Services Engineer

Basho Technologies Limited

 

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak 2.1.4 crashes with Out of Memory Error

Magnus Kessler
On 4 May 2017 at 09:56, Arulappan, Jerald (Jerald) <[hidden email]> wrote:

Hi Magnus Kessler,

 

The configuration looks good.

 

[root@server205 bin]# ./riak config effective | grep "_dir"

anti_entropy.data_dir = $(platform_data_dir)/anti_entropy

bitcask.data_root = $(platform_data_dir)/bitcask

leveldb.data_root = $(platform_data_dir)/leveldb

log.console.file = $(platform_log_dir)/console.log

log.crash.file = $(platform_log_dir)/crash.log

log.error.file = $(platform_log_dir)/error.log

platform_bin_dir = ./bin

platform_data_dir = ./data

platform_etc_dir = ./etc

platform_lib_dir = ./lib

platform_log_dir = ./log

ring.state_dir = $(platform_data_dir)/ring

search.anti_entropy.data_dir = $(platform_data_dir)/yz_anti_entropy

search.root_dir = $(platform_data_dir)/yz

search.temp_dir = $(platform_data_dir)/yz_temp

 

Regards,

Jerald



Hi Jerald,

The that fill up the logs at a very high rate are due to the use of relative file paths for platform_{bin,data,etc,lib,log}_dir. Those entries should generally contain absolute file paths, such as /var/lib/riak, as init systems may start the application from an arbitrary working directory. Please check if the errors go away after adjusting platform_data_dir.

Kind Regards,

Magnus

--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak 2.1.4 crashes with Out of Memory Error

Matthew Von-Maszewski
How much ram is in this device?  What is the ring_size setting in Riak.conf?

Thank you,
Matthew

Sent from my iPad

On May 4, 2017, at 4:08 AM, Magnus Kessler <[hidden email]> wrote:

On 4 May 2017 at 09:56, Arulappan, Jerald (Jerald) <[hidden email]> wrote:

Hi Magnus Kessler,

 

The configuration looks good.

 

[root@server205 bin]# ./riak config effective | grep "_dir"

anti_entropy.data_dir = $(platform_data_dir)/anti_entropy

bitcask.data_root = $(platform_data_dir)/bitcask

leveldb.data_root = $(platform_data_dir)/leveldb

log.console.file = $(platform_log_dir)/console.log

log.crash.file = $(platform_log_dir)/crash.log

log.error.file = $(platform_log_dir)/error.log

platform_bin_dir = ./bin

platform_data_dir = ./data

platform_etc_dir = ./etc

platform_lib_dir = ./lib

platform_log_dir = ./log

ring.state_dir = $(platform_data_dir)/ring

search.anti_entropy.data_dir = $(platform_data_dir)/yz_anti_entropy

search.root_dir = $(platform_data_dir)/yz

search.temp_dir = $(platform_data_dir)/yz_temp

 

Regards,

Jerald



Hi Jerald,

The that fill up the logs at a very high rate are due to the use of relative file paths for platform_{bin,data,etc,lib,log}_dir. Those entries should generally contain absolute file paths, such as /var/lib/riak, as init systems may start the application from an arbitrary working directory. Please check if the errors go away after adjusting platform_data_dir.

Kind Regards,

Magnus

--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...