Can't start Riak. Erlang crashes immediately. Memory problem?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't start Riak. Erlang crashes immediately. Memory problem?

prog112
So I'm running Riak on my VPS Linux machine. I've got 2GB of RAM available so I decided to go with eleveldb backend and I tuned the parameters a bit. However, I can't start my node at all. There are no errors during the riak start command but inspecting the logs show up different messages:


2013-07-11 09:21:20 =CRASH REPORT====
  crasher:
    initial call: riak_kv_index_hashtree:init/1
    pid: <0.755.0>
    registered_name: []
    exception exit: {{undef,[{eleveldb,open,["/var/lib/riak/anti_entropy/0",[{create_if_missing,true},{max_open_files,20},{write_buffer_size,10671541}]],[]},{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,426}]},{lists,foldl,3,[{file,"lists.erl"},{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},{line,368}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,225}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
    ancestors: [<0.707.0>,riak_core_vnode_sup,riak_core_sup,<0.130.0>]
    messages: []
    links: []
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 431
  neighbours:

Also:

=====
===== LOGGING STARTED Thu Jul 11 09:24:35 CEST 2013
=====
Exec: /usr/lib/riak/erts-5.9.1/bin/erlexec -boot /usr/lib/riak/releases/1.3.2/riak             -embedded -config /etc/riak/app.config             -pa /usr/lib/riak/lib/basho-patches             -args_file /etc/riak/vm.args -- console
Root: /usr/lib/riak
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:1] [async-threads:16] [kernel-poll:true]

Eshell V5.9.1  (abort with ^G)
(riak@xxx)1> /usr/lib/riak/lib/os_mon-2.2.9/priv/bin/memsup: Erlang has closed.
Erlang has closed

Anyone got any ideas for me?
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

prog112
I went even furher... Installed fresh on CentOS. Can't start at all:



=====
===== LOGGING STARTED Fri Jul 12 09:26:03 CEST 2013
=====
Node 'riak@127.0.0.1' not responding to pings.
config is OK
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot /usr/lib64/riak/releases/1.4.0/riak              -config /etc/riak/app.config             -pa /usr/lib64/riak/lib/basho-patches             -args_file /etc/riak/vm.args -- console
Root: /usr/lib64/riak
Failed to create thread: Resource temporarily unavailable (11)
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

prog112
Here it is. I tried experimenting a bit again but can't make it to work. My settings as of now: (same error)

http://wklej.org/id/1085132/ - app.config
http://wklej.org/id/1085133/ - vm.args
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

prog112

[root@s16976525 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2048         69       1978          0          0         24
-/+ buffers/cache:         45       2002
Swap:            0          0          0

The output I gave you was from riak console. Here it is again:
 riak console
config is OK
!!!!
!!!! WARNING: ulimit -n is 1024; 4096 is the recommended minimum.
!!!!
Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot /usr/lib64/riak/releases/1.4.0/riak              -config /etc/riak/app.config             -pa /usr/lib64/riak/lib/basho-patches             -args_file /etc/riak/vm.args -- console
Root: /usr/lib64/riak
Failed to create thread: Resource temporarily unavailable (11)

After rebooting it seems like no more errors with "offline node". Yet a problem with resources :( I don't know, is it really problem with threading? Could NPTL 2.12 be a problem? I'm really taking a long shot right now, as the last couple of days have been nightmare because of those riak problems.
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

prog112
Oh well, looks like after second try of startng up riak the first error returns as well:

[root@s16976525 ~]# riak console
Node 'riak@s16976525.onlinehome-server.info' not responding to pings.
config is OK
Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot /usr/lib64/riak/releases/1.4.0/riak              -config /etc/riak/app.config             -pa /usr/lib64/riak/lib/basho-patches             -args_file /etc/riak/vm.args -- console
Root: /usr/lib64/riak
Failed to create thread: Resource temporarily unavailable (11)

Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

Richard Shaw
Hi,

Please can you provide the following:

1.) Output from $ ulimit -a
2.) Output from $ uname -a
3.) Confirm which version of Riak you're installing and whether from package or source
4.) Output from $ lsof | wc -l
5.) Output from $ ps auwx

Please revert your configs to defaults, make sure for this session that ulimit is set correctly with $ ulimit -n 4096, then start Riak again

You can permanently set the correct ulimit here [0]


Thanks


On Fri, Jul 12, 2013 at 11:58 AM, prog112 <[hidden email]> wrote:
Oh well, looks like after second try of startng up riak the first error
returns as well:

[root@s16976525 ~]# riak console
Node '[hidden email]' not responding to pings.
config is OK
Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot
/usr/lib64/riak/releases/1.4.0/riak              -config
/etc/riak/app.config             -pa /usr/lib64/riak/lib/basho-patches
-args_file /etc/riak/vm.args -- console
Root: /usr/lib64/riak
Failed to create thread: Resource temporarily unavailable (11)





--
View this message in context: http://riak-users.197444.n3.nabble.com/Can-t-start-Riak-Erlang-crashes-immediately-Memory-problem-tp4028271p4028294.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

Richard Shaw
Additionally, you've only got 2GB of RAM so you could reduce the ring size to 16 and disable AAE to bring that memory usage down

/etc/riak/app.config => {ring_creation_size, 16},

/etc/riak/app.config => {anti_entropy, {off, []}},

Thanks




On Fri, Jul 12, 2013 at 4:26 PM, Richard Shaw <[hidden email]> wrote:
Hi,

Please can you provide the following:

1.) Output from $ ulimit -a
2.) Output from $ uname -a
3.) Confirm which version of Riak you're installing and whether from package or source
4.) Output from $ lsof | wc -l
5.) Output from $ ps auwx

Please revert your configs to defaults, make sure for this session that ulimit is set correctly with $ ulimit -n 4096, then start Riak again

You can permanently set the correct ulimit here [0]


Thanks


On Fri, Jul 12, 2013 at 11:58 AM, prog112 <[hidden email]> wrote:
Oh well, looks like after second try of startng up riak the first error
returns as well:

[root@s16976525 ~]# riak console
Node '[hidden email]' not responding to pings.
config is OK
Exec: /usr/lib64/riak/erts-5.9.1/bin/erlexec -boot
/usr/lib64/riak/releases/1.4.0/riak              -config
/etc/riak/app.config             -pa /usr/lib64/riak/lib/basho-patches
-args_file /etc/riak/vm.args -- console
Root: /usr/lib64/riak
Failed to create thread: Resource temporarily unavailable (11)





--
View this message in context: http://riak-users.197444.n3.nabble.com/Can-t-start-Riak-Erlang-crashes-immediately-Memory-problem-tp4028271p4028294.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

prog112
In reply to this post by prog112
So I rolled back stats to default, adding your two changes. Made sure the ring folder is gone. Changed the uname -n. Also changed number of threads in vm.args from 64 to 16 and it worked! However...

[root@s16976525 riak]# riak start
[root@s16976525 riak]# riak-admin test
Failed to read test value: {error,{insufficient_vnodes,0,need,1}}[root@s16976525 riak]#

What did I do wrong? :(

Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

Jared Morrow
Sometimes things aren't started up in time when you use 'riak-admin test' right after 'riak start'.  I've seen that happen before.  Try starting riak, waiting a minute or so and doing riak-admin test again and see if it starts writing.

-Jared


On Fri, Jul 12, 2013 at 7:54 AM, prog112 <[hidden email]> wrote:
So I rolled back stats to default, adding your two changes. Made sure the
ring folder is gone. Changed the uname -n. Also changed number of threads in
vm.args from 64 to 16 and it worked! However...

[root@s16976525 riak]# riak start
[root@s16976525 riak]# riak-admin test
Failed to read test value:
{error,{insufficient_vnodes,0,need,1}}[root@s16976525 riak]#

What did I do wrong? :(





--
View this message in context: http://riak-users.197444.n3.nabble.com/Can-t-start-Riak-Erlang-crashes-immediately-Memory-problem-tp4028271p4028298.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Can't start Riak. Erlang crashes immediately. Memory problem?

prog112
In reply to this post by Richard Shaw
Yeah it works!!!
It's taking 1.8GB of my RAM though... I'd like to get down to 1GB for now at least, I won't really have high traffic for the tests of my software. What changes do you suggest? Get ring size down to 8? Anything else I could do?

thanks for your help!