5-node low-cost, low-power Riak KV cluster -- this sound feasible?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

5-node low-cost, low-power Riak KV cluster -- this sound feasible?

lrPrentice
Hello,

I'd like to build a five-node low-cost, low-power Riak KV cluster. Use case: learning Riak,  development, testing, storage of personal data.

-- Based on pure hand-waving, I've plugged numbers that seem more than adequate for my purposes into the Riak Cluster Capacity Planning Calculator. Here's the recommendation:

To manage your estimated 100.0 thousand key/bucket pairs where bucket names are ~10 bytes, keys are ~36 bytes, values are ~97.7 KiB and you are setting aside 2.0 GiB of RAM per-node for in-memory data management within a cluster that is configured to maintain 3 replicas per key (N = 3) then Riak, using the Bitcask storage engine, will require at least:

5 nodes
6.4 MiB of RAM per node (31.9 MiB total across all nodes)
5.6 GiB of storage space per node (28.0 GiB total storage space used across all nodes)
Based on this, I'm considering the following hardware:

5 ODROID XU-4
http://www.hardkernel.com/main/products/prdt_info.php?g_code=G143452239825

Each with 8 Gb eMMC storage

This provides a 64-bit 2 GHz processor and 2 Gb RAM per day node running Ubuntu 16.04 at total cost of something under total $65000 for the cluster.

Does this sound like a feasible way to go? Any downsides?

Many thanks,

LRP


Sent from my iPad

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 5-node low-cost, low-power Riak KV cluster -- this sound feasible?

Magnus Kessler
On 27 April 2017 at 04:01, Lloyd R. Prentice <[hidden email]> wrote:
Hello,

I'd like to build a five-node low-cost, low-power Riak KV cluster. Use case: learning Riak,  development, testing, storage of personal data.

-- Based on pure hand-waving, I've plugged numbers that seem more than adequate for my purposes into the Riak Cluster Capacity Planning Calculator. Here's the recommendation:

To manage your estimated 100.0 thousand key/bucket pairs where bucket names are ~10 bytes, keys are ~36 bytes, values are ~97.7 KiB and you are setting aside 2.0 GiB of RAM per-node for in-memory data management within a cluster that is configured to maintain 3 replicas per key (N = 3) then Riak, using the Bitcask storage engine, will require at least:

5 nodes
6.4 MiB of RAM per node (31.9 MiB total across all nodes)
5.6 GiB of storage space per node (28.0 GiB total storage space used across all nodes)
Based on this, I'm considering the following hardware:

5 ODROID XU-4
http://www.hardkernel.com/main/products/prdt_info.php?g_code=G143452239825

Each with 8 Gb eMMC storage

This provides a 64-bit 2 GHz processor and 2 Gb RAM per day node running Ubuntu 16.04 at total cost of something under total $65000 for the cluster.

Does this sound like a feasible way to go? Any downsides?

Many thanks,

LRP


Hi Lloyd,

The hardware you've linked to is based on the ARM processor architecture. Please be aware that Basho does not supply prebuilt binaries for this architecture, and you would have to compile your own binaries.

If you simply want to learn how to operate a multi-node cluster you can build a so-called DEVREL release from source. This will install a configurable number of separately configured nodes on a standard OSX or Linux work station and should be sufficient to experiment with most aspects of operating a multi-node cluster.

As far as memory requirements for all but a small test work-load is concerned, we recommend to provide at least 350 MB of memory per VNode if using the leveldb backend in production. Typically, a Riak node should have between 6 and 25 VNodes (depending on ring size), and a 5 node cluster in its default configuration will have 13 or 12 VNodes per node. If you plan to use Solr based search, add at least another 1 GB for Solr. I have successfully run a test cluster with heavy Solr use on 8GB nodes.

That said, I am aware that with some extra work Riak can be built on ARM hardware, such as RaspberryPI, and has been used for demonstration purposes on this platform. You can certainly run a Riak cluster on constrained hardware if needed for testing purposes. However, if you'd like to evaluate performance characteristics, you should use a setup that mirrors an eventual production setup more closely.

I hope this helps somewhat towards finding the best platform for you.

Kind Regards,

Magnus




--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: 5-node low-cost, low-power Riak KV cluster -- this sound feasible?

Toby Corkindale-2
In reply to this post by lrPrentice
On Thu, 27 Apr 2017 at 13:02 Lloyd R. Prentice <[hidden email]> wrote:
Hello,

I'd like to build a five-node low-cost, low-power Riak KV cluster. Use case: learning Riak,  development, testing, storage of personal data.

-- Based on pure hand-waving, I've plugged numbers that seem more than adequate for my purposes into the Riak Cluster Capacity Planning Calculator. Here's the recommendation:

To manage your estimated 100.0 thousand key/bucket pairs where bucket names are ~10 bytes, keys are ~36 bytes, values are ~97.7 KiB and you are setting aside 2.0 GiB of RAM per-node for in-memory data management within a cluster that is configured to maintain 3 replicas per key (N = 3) then Riak, using the Bitcask storage engine, will require at least:

5 nodes
6.4 MiB of RAM per node (31.9 MiB total across all nodes)
5.6 GiB of storage space per node (28.0 GiB total storage space used across all nodes)
Based on this, I'm considering the following hardware:

5 ODROID XU-4
http://www.hardkernel.com/main/products/prdt_info.php?g_code=G143452239825

Each with 8 Gb eMMC storage

This provides a 64-bit 2 GHz processor and 2 Gb RAM per day node running Ubuntu 16.04 at total cost of something under total $65000 for the cluster.

Does this sound like a feasible way to go? Any downsides?


I'm not sure where you get the $65000 total figure, but that sounds extremely high!! Unless you're talking Vietnamese Dong currency or something.

Either way, why not consider running 5x Amazon AWS EC2 nodes?
Or run up a bunch of virtual machines on your own PC?

I think both of those will give you more realistic Riak results than using a bunch of ARM bigLITTLE machines which are pretty different to your typical Linux servers.

Toby


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...