max_files_limit and AAE

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

max_files_limit and AAE

Dave Brady
Hey Everyone,

We have a five-node, 128 partition cluster running 1.4.2 on Debian.

Is there a doc somewhere that explains how to size max_open_files as it applies to AAE?

I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst files in one VNode's data directory, and the boxes have plenty of RAM.

I set max_open_files in the AAE section to 3000, too, on whim after we had our first issue.  Still got these in the logs on a couple of nodes after running for less than one day:

===============
2013-11-09 11:37:12.438 [info] <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142 riak_kv/125597796958124469533129165311555572001681702912: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422> terminated with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process <0.5209.2422> with 1 neighbours exited with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302 in gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process <0.19959.2426> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}} in hashtree:new_segment_store/2 line 499 in gen_server:init_it/6 line 328
===============

Our init script has "ulimit -n 65536" in it, which I *thought* that would be high enough.  Maybe not?

I also made the necessary tweaks to /etc/pam.d/common-session*, so that /etc/security/limts.conf would be read, and that did not help.

Much obliged for any suggestions!
--
Dave Brady


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Luke Bakken
Hi Dave,

Just to confirm that the ulimit settings "stuck", could you please run riak attach and execute the following Erlang snippet?

os:cmd('ulimit -n').

The period is significant and please exit using CTRL-C twice.

Thanks!
--
Luke Bakken
CSE
[hidden email]


On Mon, Nov 11, 2013 at 11:56 AM, Dave Brady <[hidden email]> wrote:
Hey Everyone,

We have a five-node, 128 partition cluster running 1.4.2 on Debian.

Is there a doc somewhere that explains how to size max_open_files as it applies to AAE?

I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst files in one VNode's data directory, and the boxes have plenty of RAM.

I set max_open_files in the AAE section to 3000, too, on whim after we had our first issue.  Still got these in the logs on a couple of nodes after running for less than one day:

===============
2013-11-09 11:37:12.438 [info] <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142 riak_kv/125597796958124469533129165311555572001681702912: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422> terminated with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process <0.5209.2422> with 1 neighbours exited with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302 in gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process <0.19959.2426> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}} in hashtree:new_segment_store/2 line 499 in gen_server:init_it/6 line 328
===============

Our init script has "ulimit -n 65536" in it, which I *thought* that would be high enough.  Maybe not?

I also made the necessary tweaks to /etc/pam.d/common-session*, so that /etc/security/limts.conf would be read, and that did not help.

Much obliged for any suggestions!
--
Dave Brady


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Matthew Von-Maszewski
In reply to this post by Dave Brady
<base href="x-msg://42/">hmm …

128 partitions divide by 5 nodes is ~26 vnodes per server.

AAE creates a parallel number of vnodes, so your servers have ~52 vnodes each.

52 x 3,000 is 156,000 files … 156,000 > 65,536 ulimit.  Sooner or later 65,536 will be too small.  But ...

Now, the primary account method in 1.4.2 is memory size allocated based upon max_open_files.  So you have allocated 4Mbytes x 3,000 x 52 or 624,000Mbytes of RAM for leveldb.  If you truly have a 624Gbyte machine, sweet!  Otherwise, it might be time to scale back the max_open_files … and put AAE back to its default because it does not need a high max_open_files.


The tricky part to leveldb configuration is that the max_open_files parameter is per vnode / partition, not for the entire server.  This per vnode setting has caused many to over allocate, and is mostly inconsistent with every other piece of software on a Linux server.  (And caused a couple of justified rants on this user list.)  A more sane approach is coming out in Riak 2.0.

But until then, here is a spreadsheet that can help planning:


<base href="x-msg://42/">


Matthew

On Nov 11, 2013, at 2:56 PM, Dave Brady <[hidden email]> wrote:

Hey Everyone,

We have a five-node, 128 partition cluster running 1.4.2 on Debian.

Is there a doc somewhere that explains how to size max_open_files as it applies to AAE?

I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst files in one VNode's data directory, and the boxes have plenty of RAM.

I set max_open_files in the AAE section to 3000, too, on whim after we had our first issue.  Still got these in the logs on a couple of nodes after running for less than one day:

===============
2013-11-09 11:37:12.438 [info] <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142 riak_kv/125597796958124469533129165311555572001681702912: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422> terminated with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process <0.5209.2422> with 1 neighbours exited with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302 in gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process <0.19959.2426> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}} in hashtree:new_segment_store/2 line 499 in gen_server:init_it/6 line 328
===============

Our init script has "ulimit -n 65536" in it, which I *thought* that would be high enough.  Maybe not?

I also made the necessary tweaks to /etc/pam.d/common-session*, so that /etc/security/limts.conf would be read, and that did not help.

Much obliged for any suggestions!
--
Dave Brady

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

leveldb_sizing_1.4.xls (36K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Dave Brady
In reply to this post by Luke Bakken
Hi Luke,

Thanks for the fast reply!

Ok, yes, our limit is being inherited (I just raised it to 131072 after our latest issue a couple of hours ago):

$ riak attach
Remote Shell: Use "Ctrl-C a" to quit. q() or init:stop() will terminate the riak node.
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:16:16] [async-threads:0] [kernel-poll:false]

Eshell V5.9.1  (abort with ^G)
(riak@riak-01)1> os:cmd('ulimit -n').
"131072\n"
(riak@riak-01)2>

--
Dave Brady


From: "Luke Bakken" <[hidden email]>
To: "Dave Brady" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Lundi 11 Novembre 2013 21:10:54
Subject: Re: max_files_limit and AAE

Hi Dave,

Just to confirm that the ulimit settings "stuck", could you please run riak attach and execute the following Erlang snippet?

os:cmd('ulimit -n').

The period is significant and please exit using CTRL-C twice.

Thanks!
--
Luke Bakken
CSE
[hidden email]


On Mon, Nov 11, 2013 at 11:56 AM, Dave Brady <[hidden email]> wrote:
Hey Everyone,

We have a five-node, 128 partition cluster running 1.4.2 on Debian.

Is there a doc somewhere that explains how to size max_open_files as it applies to AAE?

I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst files in one VNode's data directory, and the boxes have plenty of RAM.

I set max_open_files in the AAE section to 3000, too, on whim after we had our first issue.  Still got these in the logs on a couple of nodes after running for less than one day:

===============
2013-11-09 11:37:12.438 [info] <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142 riak_kv/125597796958124469533129165311555572001681702912: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422> terminated with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process <0.5209.2422> with 1 neighbours exited with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302 in gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process <0.19959.2426> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}} in hashtree:new_segment_store/2 line 499 in gen_server:init_it/6 line 328
===============

Our init script has "ulimit -n 65536" in it, which I *thought* that would be high enough.  Maybe not?

I also made the necessary tweaks to /etc/pam.d/common-session*, so that /etc/security/limts.conf would be read, and that did not help.

Much obliged for any suggestions!
--
Dave Brady


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Dave Brady
In reply to this post by Matthew Von-Maszewski
Hi Matthew,

Yes, I *absolutely* agree that the current setting is too high.  I was just hoping to give the nodes way more than enough headroom than I thought they needed to run.  I planned to reduce the limit if I saw memory pressure.

I originally had AAE at the default of 20.  We first got those errors a couple of days ago (which was debilitating; the affected nodes went unresponsive).  I set it 3000, restarted, and less than one day later the same problem occurred (a few hours ago.)
--
Dave Brady


From: "Matthew Von-Maszewski" <[hidden email]>
To: "Dave Brady" <[hidden email]>
Cc: [hidden email]
Sent: Lundi 11 Novembre 2013 21:16:46
Subject: Re: max_files_limit and AAE

hmm …

128 partitions divide by 5 nodes is ~26 vnodes per server.

AAE creates a parallel number of vnodes, so your servers have ~52 vnodes each.

52 x 3,000 is 156,000 files … 156,000 > 65,536 ulimit.  Sooner or later 65,536 will be too small.  But ...

Now, the primary account method in 1.4.2 is memory size allocated based upon max_open_files.  So you have allocated 4Mbytes x 3,000 x 52 or 624,000Mbytes of RAM for leveldb.  If you truly have a 624Gbyte machine, sweet!  Otherwise, it might be time to scale back the max_open_files … and put AAE back to its default because it does not need a high max_open_files.


The tricky part to leveldb configuration is that the max_open_files parameter is per vnode / partition, not for the entire server.  This per vnode setting has caused many to over allocate, and is mostly inconsistent with every other piece of software on a Linux server.  (And caused a couple of justified rants on this user list.)  A more sane approach is coming out in Riak 2.0.

But until then, here is a spreadsheet that can help planning:




Matthew

On Nov 11, 2013, at 2:56 PM, Dave Brady <[hidden email]> wrote:

Hey Everyone,

We have a five-node, 128 partition cluster running 1.4.2 on Debian.

Is there a doc somewhere that explains how to size max_open_files as it applies to AAE?

I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst files in one VNode's data directory, and the boxes have plenty of RAM.

I set max_open_files in the AAE section to 3000, too, on whim after we had our first issue.  Still got these in the logs on a couple of nodes after running for less than one day:

===============
2013-11-09 11:37:12.438 [info] <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142 riak_kv/125597796958124469533129165311555572001681702912: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422> terminated with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process <0.5209.2422> with 1 neighbours exited with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302 in gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process <0.19959.2426> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}} in hashtree:new_segment_store/2 line 499 in gen_server:init_it/6 line 328
===============

Our init script has "ulimit -n 65536" in it, which I *thought* that would be high enough.  Maybe not?

I also made the necessary tweaks to /etc/pam.d/common-session*, so that /etc/security/limts.conf would be read, and that did not help.

Much obliged for any suggestions!
--
Dave Brady

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Dave Brady
Matthew,

I forgot to add "thanks" for the spreadsheet!  I will go through tomorrow (it's 10 PM here).

I have turned off AAE for the time being.

--
Dave Brady


From: "Dave Brady" <[hidden email]>
To: "Matthew Von-Maszewski" <[hidden email]>
Cc: [hidden email]
Sent: Lundi 11 Novembre 2013 21:42:32
Subject: Re: max_files_limit and AAE

Hi Matthew,

Yes, I *absolutely* agree that the current setting is too high.  I was just hoping to give the nodes way more than enough headroom than I thought they needed to run.  I planned to reduce the limit if I saw memory pressure.

I originally had AAE at the default of 20.  We first got those errors a couple of days ago (which was debilitating; the affected nodes went unresponsive).  I set it 3000, restarted, and less than one day later the same problem occurred (a few hours ago.)
--
Dave Brady


From: "Matthew Von-Maszewski" <[hidden email]>
To: "Dave Brady" <[hidden email]>
Cc: [hidden email]
Sent: Lundi 11 Novembre 2013 21:16:46
Subject: Re: max_files_limit and AAE

hmm …

128 partitions divide by 5 nodes is ~26 vnodes per server.

AAE creates a parallel number of vnodes, so your servers have ~52 vnodes each.

52 x 3,000 is 156,000 files … 156,000 > 65,536 ulimit.  Sooner or later 65,536 will be too small.  But ...

Now, the primary account method in 1.4.2 is memory size allocated based upon max_open_files.  So you have allocated 4Mbytes x 3,000 x 52 or 624,000Mbytes of RAM for leveldb.  If you truly have a 624Gbyte machine, sweet!  Otherwise, it might be time to scale back the max_open_files … and put AAE back to its default because it does not need a high max_open_files.


The tricky part to leveldb configuration is that the max_open_files parameter is per vnode / partition, not for the entire server.  This per vnode setting has caused many to over allocate, and is mostly inconsistent with every other piece of software on a Linux server.  (And caused a couple of justified rants on this user list.)  A more sane approach is coming out in Riak 2.0.

But until then, here is a spreadsheet that can help planning:




Matthew

On Nov 11, 2013, at 2:56 PM, Dave Brady <[hidden email]> wrote:

Hey Everyone,

We have a five-node, 128 partition cluster running 1.4.2 on Debian.

Is there a doc somewhere that explains how to size max_open_files as it applies to AAE?

I have max_open_files for eLevelDB set to 3000, as we have about 1500 .sst files in one VNode's data directory, and the boxes have plenty of RAM.

I set max_open_files in the AAE section to 3000, too, on whim after we had our first issue.  Still got these in the logs on a couple of nodes after running for less than one day:

===============
2013-11-09 11:37:12.438 [info] <0.857.0>@riak_kv_vnode:maybe_create_hashtrees:142 riak_kv/125597796958124469533129165311555572001681702912: unable to start index_hashtree: {error,{{badmatch,{error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}}},[{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}}
2013-11-09 11:37:12.441 [error] <0.5209.2422> gen_server <0.5209.2422> terminated with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302
2013-11-09 11:37:12.441 [error] <0.5209.2422> CRASH REPORT Process <0.5209.2422> with 1 neighbours exited with reason: no match of right hand value {error,{db_write,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/011260.log: Too many open files"}} in hashtree:flush_buffer/1 line 302 in gen_server:terminate/6 line 747
2013-11-09 11:37:12.441 [error] <0.19959.2426> CRASH REPORT Process <0.19959.2426> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: /var/lib/riak/anti_entropy/125597796958124469533129165311555572001681702912/LOCK: Too many open files"}} in hashtree:new_segment_store/2 line 499 in gen_server:init_it/6 line 328
===============

Our init script has "ulimit -n 65536" in it, which I *thought* that would be high enough.  Maybe not?

I also made the necessary tweaks to /etc/pam.d/common-session*, so that /etc/security/limts.conf would be read, and that did not help.

Much obliged for any suggestions!
--
Dave Brady

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Alexander Sicular
I'm interested to see how 2.0 fixes this. I too have been bit by the AAE killing servers problem and have had to turn it off (which is thankfully the easiest of the AAE config options). It's kind of antithesis to the easy ops proposition of Riak when a feature that is difficult to configure can kill your entire cluster. Like not just make it slow but like make it unresponsive.


@siculars
http://siculars.posthaven.com

Sent from my iRotaryPhone

> On Nov 11, 2013, at 12:59, Dave Brady <[hidden email]> wrote:
>
> I have turned off AAE for the time being.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Evan Vigil-McClanahan
AAE in 2.0 will have IO rate limiting to keep it from overwhelming disks.

On Mon, Nov 11, 2013 at 1:33 PM, Alexander Sicular <[hidden email]> wrote:

> I'm interested to see how 2.0 fixes this. I too have been bit by the AAE killing servers problem and have had to turn it off (which is thankfully the easiest of the AAE config options). It's kind of antithesis to the easy ops proposition of Riak when a feature that is difficult to configure can kill your entire cluster. Like not just make it slow but like make it unresponsive.
>
>
> @siculars
> http://siculars.posthaven.com
>
> Sent from my iRotaryPhone
>
>> On Nov 11, 2013, at 12:59, Dave Brady <[hidden email]> wrote:
>>
>> I have turned off AAE for the time being.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Dave Brady
I looked at the spreadsheet, Matthew, thanks!  It's much more comprehensive than what's on the website.

We never had any RAM issues, howver, during the incidents.  All of the machines had 35 GB or more free RAM, with no pages swapped out.

I still just don't understand what could have caused these errors.

One of the bad nodes (from yesterday's outage) has 42,000 total .sst files across its 25 VNodes (about 1,700 .sst files/VNode).

How could 65,536 filehandles got exhausted?  Would not Riak have had to open every single .sst file, *and* do <lots of other stuff>, to hit that limit?

Is there command I can use in the CLI that gives the number of open files?

And now for something only slightly different...

We have surmised that part of how the out-of-files problem appears to manifest itself caused our three recent cluster-wide outages.

We are using haproxy with the recommended config, so haproxy is set for leastconn.  What seems to have happened is that Riak continued to respond positively (at least a good part of the time) to haproxy's default aliveness check.  This caused haproxy to send all new connection requests to the bad nodes, once existing connections on the bad nodes completed.  Our cluster, in very short order, was for most intents and purposes dead.

We saw in all of our apps' logs multitudes of connection (re)attempts, which we didn't at first attribute to Riak/HAProxy. I had to get the cluster back up very quickly, so I simply did a rolling restart.

This last time (yesterday), I had a little more time to investigate.  All our apps returned to normal immediately after the second of the two identified bad nodes was restarted.

--
Dave Brady

----- Original Message -----
From: "Evan Vigil-McClanahan" <[hidden email]>
To: "Alexander Sicular" <[hidden email]>
Cc: "Dave Brady" <[hidden email]>, "riak-users" <[hidden email]>
Sent: Lundi 11 Novembre 2013 22:48:17
Subject: Re: max_files_limit and AAE

AAE in 2.0 will have IO rate limiting to keep it from overwhelming disks.

On Mon, Nov 11, 2013 at 1:33 PM, Alexander Sicular <[hidden email]> wrote:

> I'm interested to see how 2.0 fixes this. I too have been bit by the AAE killing servers problem and have had to turn it off (which is thankfully the easiest of the AAE config options). It's kind of antithesis to the easy ops proposition of Riak when a feature that is difficult to configure can kill your entire cluster. Like not just make it slow but like make it unresponsive.
>
>
> @siculars
> http://siculars.posthaven.com
>
> Sent from my iRotaryPhone
>
>> On Nov 11, 2013, at 12:59, Dave Brady <[hidden email]> wrote:
>>
>> I have turned off AAE for the time being.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Luke Bakken
Hi Dave,

You can use the lsof command to find files opened by the riak user.

--
Luke Bakken
CSE
[hidden email]


On Tue, Nov 12, 2013 at 8:08 AM, Dave Brady <[hidden email]> wrote:
I looked at the spreadsheet, Matthew, thanks!  It's much more comprehensive than what's on the website.

We never had any RAM issues, howver, during the incidents.  All of the machines had 35 GB or more free RAM, with no pages swapped out.

I still just don't understand what could have caused these errors.

One of the bad nodes (from yesterday's outage) has 42,000 total .sst files across its 25 VNodes (about 1,700 .sst files/VNode).

How could 65,536 filehandles got exhausted?  Would not Riak have had to open every single .sst file, *and* do <lots of other stuff>, to hit that limit?

Is there command I can use in the CLI that gives the number of open files?

And now for something only slightly different...

We have surmised that part of how the out-of-files problem appears to manifest itself caused our three recent cluster-wide outages.

We are using haproxy with the recommended config, so haproxy is set for leastconn.  What seems to have happened is that Riak continued to respond positively (at least a good part of the time) to haproxy's default aliveness check.  This caused haproxy to send all new connection requests to the bad nodes, once existing connections on the bad nodes completed.  Our cluster, in very short order, was for most intents and purposes dead.

We saw in all of our apps' logs multitudes of connection (re)attempts, which we didn't at first attribute to Riak/HAProxy. I had to get the cluster back up very quickly, so I simply did a rolling restart.

This last time (yesterday), I had a little more time to investigate.  All our apps returned to normal immediately after the second of the two identified bad nodes was restarted.

--
Dave Brady

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Shane McEwan-2
In reply to this post by Dave Brady
On 12/11/13 16:08, Dave Brady wrote:
> Is there command I can use in the CLI that gives the number of open files?

sudo ls -1 /proc/`pidof beam.smp`/fd | wc -l

You say you have 42,000 .sst files. Does that include AAE? If not, AAE
could push you over the edge.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Dave Brady
In reply to this post by Luke Bakken
Hi Luke,

Yes, though that's kind of intensive.

Thanks!

--
Dave Brady


From: "Luke Bakken" <[hidden email]>
To: "Dave Brady" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Mardi 12 Novembre 2013 17:18:40
Subject: Re: max_files_limit and AAE

Hi Dave,

You can use the lsof command to find files opened by the riak user.

--
Luke Bakken
CSE
[hidden email]


On Tue, Nov 12, 2013 at 8:08 AM, Dave Brady <[hidden email]> wrote:
I looked at the spreadsheet, Matthew, thanks!  It's much more comprehensive than what's on the website.

We never had any RAM issues, howver, during the incidents.  All of the machines had 35 GB or more free RAM, with no pages swapped out.

I still just don't understand what could have caused these errors.

One of the bad nodes (from yesterday's outage) has 42,000 total .sst files across its 25 VNodes (about 1,700 .sst files/VNode).

How could 65,536 filehandles got exhausted?  Would not Riak have had to open every single .sst file, *and* do <lots of other stuff>, to hit that limit?

Is there command I can use in the CLI that gives the number of open files?

And now for something only slightly different...

We have surmised that part of how the out-of-files problem appears to manifest itself caused our three recent cluster-wide outages.

We are using haproxy with the recommended config, so haproxy is set for leastconn.  What seems to have happened is that Riak continued to respond positively (at least a good part of the time) to haproxy's default aliveness check.  This caused haproxy to send all new connection requests to the bad nodes, once existing connections on the bad nodes completed.  Our cluster, in very short order, was for most intents and purposes dead.

We saw in all of our apps' logs multitudes of connection (re)attempts, which we didn't at first attribute to Riak/HAProxy. I had to get the cluster back up very quickly, so I simply did a rolling restart.

This last time (yesterday), I had a little more time to investigate.  All our apps returned to normal immediately after the second of the two identified bad nodes was restarted.

--
Dave Brady


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: max_files_limit and AAE

Dave Brady
In reply to this post by Shane McEwan-2
Thanks, Shane!

No, it doesn't include AAE.

I'm leaving AAE off for now, anyway.

I've got ulimit at 131072 now, and that (very quick-to-run) command of yours shows about 32,000, so it seems we've got even headroom now.

--
Dave Brady

----- Original Message -----
From: "Shane McEwan" <[hidden email]>
To: [hidden email]
Sent: Mardi 12 Novembre 2013 17:33:01
Subject: Re: max_files_limit and AAE

On 12/11/13 16:08, Dave Brady wrote:
> Is there command I can use in the CLI that gives the number of open files?

sudo ls -1 /proc/`pidof beam.smp`/fd | wc -l

You say you have 42,000 .sst files. Does that include AAE? If not, AAE
could push you over the edge.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com