Multiple nodes leaving cluster

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Multiple nodes leaving cluster

Daniel Miller

Hi Riak Users,

In the documentation for riak-admin cluster leave it says “You can stage multiple leave command before planning/committing.” This implies that it is safe to stage multiple nodes leaving the cluster simultaneously. Is that true? Will all data in the cluster be continuously available during the removal period if, for example, I setup and commit a plan for 3 nodes to be leave a 9-node cluster (assuming there is enough space for the data on the remaining 6 nodes)?

I had asked a similar question on IRC a couple weeks ago. In that case I was asking about replacing multiple nodes simultaneously using riak-admin cluster replace. The answer I got there left some doubt in my mind as to whether it is safe (i.e., will not result in a period data availability) to have multiple nodes leaving the cluster at once. The documentation for replace implies that it is safe to replace multiple nodes simultaneously as well: “You can stage multiple replace actions before planning/committing.”

Note that I am not asking about force-remove or force-replace, which I would expect to result in permanent data loss if multiple nodes are force-removed/replaced simultaneously.

My cluster is running Riak 2.1.1 with standard nval of 3.

Thanks!
Daniel


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Multiple nodes leaving cluster

Shaun McVey
Hi Daniel,

It should be safe, yes.  If you run the `cluster leave` commands, then run `cluster plan` (but don't yet run `cluster commit`), the output will show if there's any violations in spreading each of your 3 copies around the cluster.  If it does provide a warning, you can `cluster clear` and backout the changes.  If not, then you can `cluster commit` and the data will be moved from the leaving nodes to the rest of the cluster.  Whether or not you see any violations (i.e. more than one copy of data being stored on the same physical server) will depend on a combination of number of nodes and the ring size.

Shaun

On Tue, Feb 7, 2017 at 11:25 PM, Daniel Miller <[hidden email]> wrote:

Hi Riak Users,

In the documentation for riak-admin cluster leave it says “You can stage multiple leave command before planning/committing.” This implies that it is safe to stage multiple nodes leaving the cluster simultaneously. Is that true? Will all data in the cluster be continuously available during the removal period if, for example, I setup and commit a plan for 3 nodes to be leave a 9-node cluster (assuming there is enough space for the data on the remaining 6 nodes)?

I had asked a similar question on IRC a couple weeks ago. In that case I was asking about replacing multiple nodes simultaneously using riak-admin cluster replace. The answer I got there left some doubt in my mind as to whether it is safe (i.e., will not result in a period data availability) to have multiple nodes leaving the cluster at once. The documentation for replace implies that it is safe to replace multiple nodes simultaneously as well: “You can stage multiple replace actions before planning/committing.”

Note that I am not asking about force-remove or force-replace, which I would expect to result in permanent data loss if multiple nodes are force-removed/replaced simultaneously.

My cluster is running Riak 2.1.1 with standard nval of 3.

Thanks!
Daniel


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Multiple nodes leaving cluster

Magnus Kessler
In reply to this post by Daniel Miller
On 7 February 2017 at 23:25, Daniel Miller <[hidden email]> wrote:

Hi Riak Users,

In the documentation for riak-admin cluster leave it says “You can stage multiple leave command before planning/committing.” This implies that it is safe to stage multiple nodes leaving the cluster simultaneously. Is that true? Will all data in the cluster be continuously available during the removal period if, for example, I setup and commit a plan for 3 nodes to be leave a 9-node cluster (assuming there is enough space for the data on the remaining 6 nodes)?

I had asked a similar question on IRC a couple weeks ago. In that case I was asking about replacing multiple nodes simultaneously using riak-admin cluster replace. The answer I got there left some doubt in my mind as to whether it is safe (i.e., will not result in a period data availability) to have multiple nodes leaving the cluster at once. The documentation for replace implies that it is safe to replace multiple nodes simultaneously as well: “You can stage multiple replace actions before planning/committing.”

Note that I am not asking about force-remove or force-replace, which I would expect to result in permanent data loss if multiple nodes are force-removed/replaced simultaneously.

My cluster is running Riak 2.1.1 with standard nval of 3.

Thanks!
Daniel


Hi Daniel,

Yes, staging several riak-admin cluster leave steps before  riak-admin cluster commit is safe. The leaving nodes will perform an ownership handoff of all their partitions to other nodes in the cluster before shutting themselves down. While this is happening, these nodes remain in the cluster as fully functional nodes. The same is true for  riak-admin cluster replace.

The reason we recommend staging of multiple leave or join operations is that this minimises reshuffling data throughout the cluster. Only set of ownership handoffs needs to happen, whereas if you were to perform several consecutive leave or join operations, on each commit a substantial amount of handoffs throughout the cluster is needed.

Kind Regards,

Magnus


--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Multiple nodes leaving cluster

Daniel Miller
Thanks Magnus and Shaun, that's very helpful.

Daniel

On Wed, Feb 8, 2017 at 4:00 AM, Magnus Kessler <[hidden email]> wrote:
On 7 February 2017 at 23:25, Daniel Miller <[hidden email]> wrote:

Hi Riak Users,

In the documentation for riak-admin cluster leave it says “You can stage multiple leave command before planning/committing.” This implies that it is safe to stage multiple nodes leaving the cluster simultaneously. Is that true? Will all data in the cluster be continuously available during the removal period if, for example, I setup and commit a plan for 3 nodes to be leave a 9-node cluster (assuming there is enough space for the data on the remaining 6 nodes)?

I had asked a similar question on IRC a couple weeks ago. In that case I was asking about replacing multiple nodes simultaneously using riak-admin cluster replace. The answer I got there left some doubt in my mind as to whether it is safe (i.e., will not result in a period data availability) to have multiple nodes leaving the cluster at once. The documentation for replace implies that it is safe to replace multiple nodes simultaneously as well: “You can stage multiple replace actions before planning/committing.”

Note that I am not asking about force-remove or force-replace, which I would expect to result in permanent data loss if multiple nodes are force-removed/replaced simultaneously.

My cluster is running Riak 2.1.1 with standard nval of 3.

Thanks!
Daniel


Hi Daniel,

Yes, staging several riak-admin cluster leave steps before  riak-admin cluster commit is safe. The leaving nodes will perform an ownership handoff of all their partitions to other nodes in the cluster before shutting themselves down. While this is happening, these nodes remain in the cluster as fully functional nodes. The same is true for  riak-admin cluster replace.

The reason we recommend staging of multiple leave or join operations is that this minimises reshuffling data throughout the cluster. Only set of ownership handoffs needs to happen, whereas if you were to perform several consecutive leave or join operations, on each commit a substantial amount of handoffs throughout the cluster is needed.

Kind Regards,

Magnus


--
Magnus Kessler
Client Services Engineer
Basho Technologies Limited

Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...