Minimal number of nodes for production

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Minimal number of nodes for production

Tom Zeng
Hi list,

We have a production installation with only 3 nodes and running on 1.2.1.  I'd appreciate to get some  facts to convince IT to increase the number of nodes to 7 and upgrade to 1.3.  I heard people from Basho mentioned ideally 7 nodes for production a couple of time, can someone explain why 7, is 4, or 5 nodes good enough?

Also on the 3 three nodes, the file size for the bitcask directory very quite a bit: 21GB, 14GB, and 20GB. Could the node with only 14GB missing something or it's expected to have such big difference?

Thanks,
Tom

--
Tom Zeng
Director of Engineering
Intridea, Inc. | www.intridea.com
(o) 888.968.4332 x519
(c) 240-643-8728

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Minimal number of nodes for production

Eric Redmond-2


On Apr 10, 2013, at 2:26 PM, Tom Zeng <[hidden email]> wrote:

Hi list,

We have a production installation with only 3 nodes and running on 1.2.1.  I'd appreciate to get some  facts to convince IT to increase the number of nodes to 7 and upgrade to 1.3.  I heard people from Basho mentioned ideally 7 nodes for production a couple of time, can someone explain why 7, is 4, or 5 nodes good enough?

I'm not sure where you heard the number 7 as a minimum, unless if was for a specific use-case. In general the minimum recommended number is 5 nodes.

Running with only 3 nodes isn't a great idea. Since a core purpose of Riak is to remain available in the face of outages, 3 will not support any outage. Less than 3 is lower than the default replication value (N=3). This is so important, in fact, that we recommend 5 solely to act as a buffer in the case where 1 of the 5 is down, the remaining 4 is dangerously close to the inflexible 3 node number. Even if you do not upgrade to 1.3, you really need to have at least 5 nodes.

There are many benefits to upgrading to 1.3, but one of the most compelling from an operations point of view is active anti-entropy (AAE). Rather than waiting on read-repair to fix inconsistent values (which is passive), AAE routinely attempts to keep all node values in sync. This can be a godsend if a node goes down, since you don't need to fore read-repair when you bring the node back up by reading every key... you just let your cluster actively self-heal.


Also on the 3 three nodes, the file size for the bitcask directory very quite a bit: 21GB, 14GB, and 20GB. Could the node with only 14GB missing something or it's expected to have such big difference?

There are several reasons sizes could be different. Values are not yet/ever replicated (based on your N and W values). Files may have not been compacted. Some keys have been deleted but not yet reaped...

Thanks,
Tom

--
Tom Zeng
Director of Engineering
Intridea, Inc. | www.intridea.com
(o) 888.968.4332 x519
(c) 240-643-8728
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Minimal number of nodes for production

Tom Zeng
Thanks Eric for the info, that's very helpful.  7 was mentioned at the last Riak DC meetup. not as the minimal but for better performance, when I was chatting with a couple of Basho devs about performance benchmarking, and about Riak is quite a bit slower on single node against Mongo.


On Wed, Apr 10, 2013 at 5:56 PM, Eric Redmond <[hidden email]> wrote:


On Apr 10, 2013, at 2:26 PM, Tom Zeng <[hidden email]> wrote:

Hi list,

We have a production installation with only 3 nodes and running on 1.2.1.  I'd appreciate to get some  facts to convince IT to increase the number of nodes to 7 and upgrade to 1.3.  I heard people from Basho mentioned ideally 7 nodes for production a couple of time, can someone explain why 7, is 4, or 5 nodes good enough?

I'm not sure where you heard the number 7 as a minimum, unless if was for a specific use-case. In general the minimum recommended number is 5 nodes.

Running with only 3 nodes isn't a great idea. Since a core purpose of Riak is to remain available in the face of outages, 3 will not support any outage. Less than 3 is lower than the default replication value (N=3). This is so important, in fact, that we recommend 5 solely to act as a buffer in the case where 1 of the 5 is down, the remaining 4 is dangerously close to the inflexible 3 node number. Even if you do not upgrade to 1.3, you really need to have at least 5 nodes.

There are many benefits to upgrading to 1.3, but one of the most compelling from an operations point of view is active anti-entropy (AAE). Rather than waiting on read-repair to fix inconsistent values (which is passive), AAE routinely attempts to keep all node values in sync. This can be a godsend if a node goes down, since you don't need to fore read-repair when you bring the node back up by reading every key... you just let your cluster actively self-heal.


Also on the 3 three nodes, the file size for the bitcask directory very quite a bit: 21GB, 14GB, and 20GB. Could the node with only 14GB missing something or it's expected to have such big difference?

There are several reasons sizes could be different. Values are not yet/ever replicated (based on your N and W values). Files may have not been compacted. Some keys have been deleted but not yet reaped...

Thanks,
Tom

--
Tom Zeng
Director of Engineering
Intridea, Inc. | www.intridea.com
(o) <a href="tel:888.968.4332%20x519" value="+18889684332" target="_blank">888.968.4332 x519
(c) <a href="tel:240-643-8728" value="+12406438728" target="_blank">240-643-8728
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Minimal number of nodes for production

Eric Redmond-2
I certainly won't get drawn into a public debate about which database is faster per node, but I would recommend you never take such claims at face value.

Eric


On Apr 10, 2013, at 3:02 PM, Tom Zeng <[hidden email]> wrote:

Thanks Eric for the info, that's very helpful.  7 was mentioned at the last Riak DC meetup. not as the minimal but for better performance, when I was chatting with a couple of Basho devs about performance benchmarking, and about Riak is quite a bit slower on single node against Mongo.


On Wed, Apr 10, 2013 at 5:56 PM, Eric Redmond <[hidden email]> wrote:


On Apr 10, 2013, at 2:26 PM, Tom Zeng <[hidden email]> wrote:

Hi list,

We have a production installation with only 3 nodes and running on 1.2.1.  I'd appreciate to get some  facts to convince IT to increase the number of nodes to 7 and upgrade to 1.3.  I heard people from Basho mentioned ideally 7 nodes for production a couple of time, can someone explain why 7, is 4, or 5 nodes good enough?

I'm not sure where you heard the number 7 as a minimum, unless if was for a specific use-case. In general the minimum recommended number is 5 nodes.

Running with only 3 nodes isn't a great idea. Since a core purpose of Riak is to remain available in the face of outages, 3 will not support any outage. Less than 3 is lower than the default replication value (N=3). This is so important, in fact, that we recommend 5 solely to act as a buffer in the case where 1 of the 5 is down, the remaining 4 is dangerously close to the inflexible 3 node number. Even if you do not upgrade to 1.3, you really need to have at least 5 nodes.

There are many benefits to upgrading to 1.3, but one of the most compelling from an operations point of view is active anti-entropy (AAE). Rather than waiting on read-repair to fix inconsistent values (which is passive), AAE routinely attempts to keep all node values in sync. This can be a godsend if a node goes down, since you don't need to fore read-repair when you bring the node back up by reading every key... you just let your cluster actively self-heal.


Also on the 3 three nodes, the file size for the bitcask directory very quite a bit: 21GB, 14GB, and 20GB. Could the node with only 14GB missing something or it's expected to have such big difference?

There are several reasons sizes could be different. Values are not yet/ever replicated (based on your N and W values). Files may have not been compacted. Some keys have been deleted but not yet reaped...

Thanks,
Tom

--
Tom Zeng
Director of Engineering
Intridea, Inc. | www.intridea.com
(o) <a href="tel:888.968.4332%20x519" value="+18889684332" target="_blank">888.968.4332 x519
(c) <a href="tel:240-643-8728" value="+12406438728" target="_blank">240-643-8728
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Minimal number of nodes for production

Tom Zeng
That's good recommendation, thanks:-).  I will have a 10-15 node environment up for R&D soon, will do some benchmarking in that environment. 

On Wed, Apr 10, 2013 at 6:17 PM, Eric Redmond <[hidden email]> wrote:
I certainly won't get drawn into a public debate about which database is faster per node, but I would recommend you never take such claims at face value.

Eric


On Apr 10, 2013, at 3:02 PM, Tom Zeng <[hidden email]> wrote:

Thanks Eric for the info, that's very helpful.  7 was mentioned at the last Riak DC meetup. not as the minimal but for better performance, when I was chatting with a couple of Basho devs about performance benchmarking, and about Riak is quite a bit slower on single node against Mongo.


On Wed, Apr 10, 2013 at 5:56 PM, Eric Redmond <[hidden email]> wrote:


On Apr 10, 2013, at 2:26 PM, Tom Zeng <[hidden email]> wrote:

Hi list,

We have a production installation with only 3 nodes and running on 1.2.1.  I'd appreciate to get some  facts to convince IT to increase the number of nodes to 7 and upgrade to 1.3.  I heard people from Basho mentioned ideally 7 nodes for production a couple of time, can someone explain why 7, is 4, or 5 nodes good enough?

I'm not sure where you heard the number 7 as a minimum, unless if was for a specific use-case. In general the minimum recommended number is 5 nodes.

Running with only 3 nodes isn't a great idea. Since a core purpose of Riak is to remain available in the face of outages, 3 will not support any outage. Less than 3 is lower than the default replication value (N=3). This is so important, in fact, that we recommend 5 solely to act as a buffer in the case where 1 of the 5 is down, the remaining 4 is dangerously close to the inflexible 3 node number. Even if you do not upgrade to 1.3, you really need to have at least 5 nodes.

There are many benefits to upgrading to 1.3, but one of the most compelling from an operations point of view is active anti-entropy (AAE). Rather than waiting on read-repair to fix inconsistent values (which is passive), AAE routinely attempts to keep all node values in sync. This can be a godsend if a node goes down, since you don't need to fore read-repair when you bring the node back up by reading every key... you just let your cluster actively self-heal.


Also on the 3 three nodes, the file size for the bitcask directory very quite a bit: 21GB, 14GB, and 20GB. Could the node with only 14GB missing something or it's expected to have such big difference?

There are several reasons sizes could be different. Values are not yet/ever replicated (based on your N and W values). Files may have not been compacted. Some keys have been deleted but not yet reaped...

Thanks,
Tom

--
Tom Zeng
Director of Engineering
Intridea, Inc. | www.intridea.com
(o) <a href="tel:888.968.4332%20x519" value="+18889684332" target="_blank">888.968.4332 x519
(c) <a href="tel:240-643-8728" value="+12406438728" target="_blank">240-643-8728
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Minimal number of nodes for production

Daniel Iwan
With 3 nodes only if one of the nodes is down I should still be able to read write if I'm doing that with default setting (quorum). Am I wrong.
I appreciate there is a case that SHA key space does not divide equally by 3 so there is a slim possibility that  replicas for some keys would be on the same physical hardware. But still, 99% of keys should be available. Right?

Daniel
Reply | Threaded
Open this post in threaded view
|

Re: Minimal number of nodes for production

Christian Dahlqvist
Hi Daniel,

If you have 3 nodes in the cluster, you should not lose any data if one node goes down, but you may experience that some records, for which 2 replicas are gone, will return false not founds before read-repair can fix it. If you therefore retry whenever you do not find a key that you expect, you should be fine.

The other issue with having few nodes in the cluster is that each of the remaining nodes will need to manage a significant number of fallback partitions during node failure, which can quickly increase the load on these nodes. There is also a risk that the fallback partitions may not be divided equally between the remaining two nodes, leading to the load on one node increasing even further.

It is in order to avoid this and be able to ensure that nodes are reasonably evenly loaded in case of a failure that we recommend at least 5 nodes in any production cluster.

Best regards,

Christian



On 11 Apr 2013, at 12:01, ivenhov <[hidden email]> wrote:

With 3 nodes only if one of the nodes is down I should still be able to read
write if I'm doing that with default setting (quorum). Am I wrong.
I appreciate there is a case that SHA key space does not divide equally by 3
so there is a slim possibility that  replicas for some keys would be on the
same physical hardware. But still, 99% of keys should be available. Right?

Daniel



--
View this message in context: http://riak-users.197444.n3.nabble.com/Minimal-number-of-nodes-for-production-tp4027625p4027635.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com