Memory Errors when using RiakCS

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory Errors when using RiakCS

Ben Rudolph
Hi,

We've been trying to setup RiakCS for storing files on our website, http://commcarehq.org/. It's been difficult for us to deploy because we've been having issues with our servers not being able to start Riak due to memory errors. Here's what a traceback looks like when I try and start Riak with `riak-console`:

config is OK
-config /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config -args_file /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args -vm_args /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
Exec:  /usr/lib/riak-cs/erts-5.10.3/bin/erlexec -boot /usr/lib/riak-cs/releases/2.0.1/riak-cs               -config /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config -args_file /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args -vm_args /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args              -pa /usr/lib/riak-cs/lib/basho-patches -- console
Root: /usr/lib/riak-cs
Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true] [frame-pointer]

=INFO REPORT==== 23-Nov-2015::18:11:18 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
=INFO REPORT==== 23-Nov-2015::18:11:18 ===
opening log file: "/var/log/riak-cs/access.log.2015_11_23_18"
Eshell V5.10.3  (abort with ^G)
(**)1> [os_mon] memory supervisor port (memsup): Erlang has closed 
 
The machines we are trying to deploy to have 8GB of RAM and also running a few other services, but have 3-4GB of RAM available. Are there any configuration params we can use so it consumes less memory? I've taken a look at the `system_memory_high_watermark` which is the error we see (http://docs.basho.com/riak/latest/ops/running/recovery/errors/). Our configs can be found here in case that's useful:


Any insight would be useful! Thank you!

Ben


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Memory Errors when using RiakCS

Charlie Voiselle
Ben:

Awesome!  Happy that that got you sorted out.  

-cv


On Nov 24, 2015, at 11:57 AM, Ben Rudolph <[hidden email]> wrote:

Right you are:
2015-11-23 18:11:19.260 [warning] <0.142.0>@riak_cs_app:check_bucket_props:128 Unable to verify moss.access bucket settings (disconnected).
2015-11-23 18:11:19.261 [warning] <0.142.0>@riak_cs_app:check_bucket_props:128 Unable to verify moss.storage bucket settings (disconnected).
2015-11-23 18:11:19.261 [warning] <0.142.0>@riak_cs_app:check_bucket_props:128 Unable to verify moss.buckets bucket settings (disconnected).
2015-11-23 18:11:19.261 [error] <0.142.0>@riak_cs_app:sanity_check:70 Could not verify bucket properties. Error was disconnected.
2015-11-23 18:11:19.261 [error] <0.141.0> CRASH REPORT Process <0.141.0> with 0 neighbours exited with reason: {error_verifying_props,{riak_cs_app,start,[normal,[]]}} in application_master:init/4 line 133
2015-11-23 18:11:19.262 [info] <0.7.0> Application riak_cs exited with reason: {error_verifying_props,{riak_cs_app,start,[normal,[]]}}

Dang, sorry I think I was misled by the `config OK` signal at the beginning of the error.

On Tue, Nov 24, 2015 at 11:40 AM, Charlie Voiselle <[hidden email]> wrote:
Ben:

It is not uncommon to see the 
[os_mon] memory supervisor port (memsup): Erlang has closed 
message when the Erlang VM shuts down.   Would it be possible for you to share your console.log file to the list?  There is probably another configuration error causing your riak-cs nodes to shut down automatically.   Misconfigured default bucket types is one such problem that will cause a riak-cs node to start and then stop immediately.

Thanks,
Charlie Voiselle
Basho, Client Services.


On Nov 23, 2015, at 1:30 PM, Ben Rudolph <[hidden email]> wrote:

Hi,

We've been trying to setup RiakCS for storing files on our website, http://commcarehq.org/. It's been difficult for us to deploy because we've been having issues with our servers not being able to start Riak due to memory errors. Here's what a traceback looks like when I try and start Riak with `riak-console`:

config is OK
-config /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config -args_file /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args -vm_args /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
Exec:  /usr/lib/riak-cs/erts-5.10.3/bin/erlexec -boot /usr/lib/riak-cs/releases/2.0.1/riak-cs               -config /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config -args_file /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args -vm_args /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args              -pa /usr/lib/riak-cs/lib/basho-patches -- console
Root: /usr/lib/riak-cs
Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:8:8] [async-threads:64] [kernel-poll:true] [frame-pointer]

=INFO REPORT==== 23-Nov-2015::18:11:18 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
=INFO REPORT==== 23-Nov-2015::18:11:18 ===
opening log file: "/var/log/riak-cs/access.log.2015_11_23_18"
Eshell V5.10.3  (abort with ^G)
(**)1> [os_mon] memory supervisor port (memsup): Erlang has closed 
 
The machines we are trying to deploy to have 8GB of RAM and also running a few other services, but have 3-4GB of RAM available. Are there any configuration params we can use so it consumes less memory? I've taken a look at the `system_memory_high_watermark` which is the error we see (http://docs.basho.com/riak/latest/ops/running/recovery/errors/). Our configs can be found here in case that's useful:


Any insight would be useful! Thank you!

Ben



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Memory Errors when using RiakCS

Kota Uenishi
In reply to this post by Ben Rudolph
Hi Ben,
The memory size of your nodes looks rather small. I wrote a section
for such environment to design right configurations. Hope you could
take a look.

https://github.com/basho/riak_cs/blob/develop/RELEASE-NOTES.md#upgrading-riak---step-3-review-memory-size

On Tue, Nov 24, 2015 at 3:30 AM, Ben Rudolph <[hidden email]> wrote:

> Hi,
>
> We've been trying to setup RiakCS for storing files on our website,
> http://commcarehq.org/. It's been difficult for us to deploy because we've
> been having issues with our servers not being able to start Riak due to
> memory errors. Here's what a traceback looks like when I try and start Riak
> with `riak-console`:
>
>> config is OK
>> -config /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config
>> -args_file /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
>> -vm_args /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
>> Exec:  /usr/lib/riak-cs/erts-5.10.3/bin/erlexec -boot
>> /usr/lib/riak-cs/releases/2.0.1/riak-cs               -config
>> /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config -args_file
>> /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args -vm_args
>> /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
>> -pa /usr/lib/riak-cs/lib/basho-patches -- console
>> Root: /usr/lib/riak-cs
>> Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:8:8]
>> [async-threads:64] [kernel-poll:true] [frame-pointer]
>>
>> =INFO REPORT==== 23-Nov-2015::18:11:18 ===
>>     alarm_handler: {set,{system_memory_high_watermark,[]}}
>> =INFO REPORT==== 23-Nov-2015::18:11:18 ===
>> opening log file: "/var/log/riak-cs/access.log.2015_11_23_18"
>> Eshell V5.10.3  (abort with ^G)
>> (**)1> [os_mon] memory supervisor port (memsup): Erlang has closed
>>
>>
>
> The machines we are trying to deploy to have 8GB of RAM and also running a
> few other services, but have 3-4GB of RAM available. Are there any
> configuration params we can use so it consumes less memory? I've taken a
> look at the `system_memory_high_watermark` which is the error we see
> (http://docs.basho.com/riak/latest/ops/running/recovery/errors/). Our
> configs can be found here in case that's useful:
>
> https://github.com/dimagi/commcarehq-ansible/tree/riak/ansible/roles/riakcs/templates
>
> Any insight would be useful! Thank you!
>
> Ben
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



--
Kota UENISHI / @kuenishi
Basho Japan KK

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Memory Errors when using RiakCS

Ben Rudolph
Thanks Kota, this is a great resource.

On Wed, Nov 25, 2015 at 4:24 AM, Kota Uenishi <[hidden email]> wrote:
Hi Ben,
The memory size of your nodes looks rather small. I wrote a section
for such environment to design right configurations. Hope you could
take a look.

https://github.com/basho/riak_cs/blob/develop/RELEASE-NOTES.md#upgrading-riak---step-3-review-memory-size

On Tue, Nov 24, 2015 at 3:30 AM, Ben Rudolph <[hidden email]> wrote:
> Hi,
>
> We've been trying to setup RiakCS for storing files on our website,
> http://commcarehq.org/. It's been difficult for us to deploy because we've
> been having issues with our servers not being able to start Riak due to
> memory errors. Here's what a traceback looks like when I try and start Riak
> with `riak-console`:
>
>> config is OK
>> -config /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config
>> -args_file /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
>> -vm_args /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
>> Exec:  /usr/lib/riak-cs/erts-5.10.3/bin/erlexec -boot
>> /usr/lib/riak-cs/releases/2.0.1/riak-cs               -config
>> /var/lib/riak-cs/generated.configs/app.2015.11.23.18.11.17.config -args_file
>> /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args -vm_args
>> /var/lib/riak-cs/generated.configs/vm.2015.11.23.18.11.17.args
>> -pa /usr/lib/riak-cs/lib/basho-patches -- console
>> Root: /usr/lib/riak-cs
>> Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:8:8]
>> [async-threads:64] [kernel-poll:true] [frame-pointer]
>>
>> =INFO REPORT==== 23-Nov-2015::18:11:18 ===
>>     alarm_handler: {set,{system_memory_high_watermark,[]}}
>> =INFO REPORT==== 23-Nov-2015::18:11:18 ===
>> opening log file: "/var/log/riak-cs/access.log.2015_11_23_18"
>> Eshell V5.10.3  (abort with ^G)
>> (**)1> [os_mon] memory supervisor port (memsup): Erlang has closed
>>
>>
>
> The machines we are trying to deploy to have 8GB of RAM and also running a
> few other services, but have 3-4GB of RAM available. Are there any
> configuration params we can use so it consumes less memory? I've taken a
> look at the `system_memory_high_watermark` which is the error we see
> (http://docs.basho.com/riak/latest/ops/running/recovery/errors/). Our
> configs can be found here in case that's useful:
>
> https://github.com/dimagi/commcarehq-ansible/tree/riak/ansible/roles/riakcs/templates
>
> Any insight would be useful! Thank you!
>
> Ben
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



--
Kota UENISHI / @kuenishi
Basho Japan KK


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com