riak_core questions

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

riak_core questions

Dmitry Demeshchuk
Greetings. I'm building an application on top of riak_core and I got
several questions about how all this stuff works.

1. When a node goes down, I need to force its exclusion from the
cluster, so the vnodes will be redistributed between the working
nodes.

As far as I understad, the easiest way to do it is something like that:


==================================================
...
riak_core_node_watcher_events:add_handler(failover_module, []),
...
==================================================
-module(failover_module).
-behaviour(gen_event).
....
handle_event({service_update, [Service]}, State) ->
    NewNodes = riak_core_node_watcher:nodes(Service),
    OldNodes = riak_core_ring:all_members(riak_core_ring_manager:get_my_ring()),
    [Node2Remove] = OldNodes -- NewNodes,
    riak_core:remove_from_cluster(Node2Remove),
    {ok, State}
======================================================

Isn't there an easier way than that? Also, it's not obvious to me if I
should execute this code on each node, or when it's executed all the
nodes in the ring are automatically updated.

2. When a node goes up, there are 2 possible scenarios: if the node
did exist in the cluster before, we have to add it manually (which is
okay); and if the node did exist in the cluster before going down, it
gets the master node from the ring file and makes rpc:call which adds
this node back to the cluster.

3. Each vnode has an attached process that does some stuff. When the
cluster is re-arranged (and vnodes are re-distributed among Erlang
nodes), I still need those attached processes to finish all the stuff
they are doing, and only then start moving the corresponding vnode to
the new node. Is it enough just to wait for the corresponding message
{you_can_move_vnode_now} at riak_more_vnode_module:terminate/2, or it
needs to be done in some other way?

4. Finally, I'm a bit concerned about what happens to the cluster if
the master node goes down. Probably, we cannot remove master node from
the cluster. Or riak_core automatically elects the new master node in
such a case (unlikely, I haven't found any gen_leader-like stuff).
What's the best way to handle such situations?

Thank you.


--
Best regards,
Dmitry Demeshchuk

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riak_core questions

Justin Sheehy
Hi, Dmitry.

A couple of suggestions...

The reason that you're not seeing an easy way to automatically have nodes be added or removed from the cluster upon going down or coming up is that we recommend strongly against such behavior.

The idea is that intentional (administrative) outages are very different in nature from unintentional and potentially transitory outages. We have explicit administrative commands such as "join" and "leave" for the administrative cases, making it very easy to add or remove hosts to a cluster. When a node is unreachable, you often can't automatically tell whether it is a host problem or a network problem and can't automatically tell if it is a long-term or short-term outage. This is why mechanisms such as quorums and hinted handoff exist: to ensure proper operation of the cluster as a whole throughout such outages. Consider the case where you have a network problem such that several of your nodes lose visibility to each other for brief and distinct periods of time. If nodes are auto-added and auto-removed then you will have quite a bit of churn and potentially a very harmful feedback scenario. Instead of auto-adding and auto-removing, consider using things like
riak_core_node_watcher to decide which nodes to interact with on a per-operation basis.

I'm also not sure what you mean by "if the master node goes down" since in most riak_core applications there is no master node. Of course you can create such a mechanism if you need it, but (e.g.) Riak KV and the accompanying applications do not have any notion of a master node and thus do not have any such concern.

I hope that this is useful.

Best regards,

-Justin



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riak_core questions

Dmitry Demeshchuk
Hi, Justin.

Unfortunately, the application should behave like that, though I don't
like it myself. During any outage, we should be completely sure that
all the vnodes are up, and we cannot use any kind of replication in
our case. It is just the way to ensure that we don't access the same
resource at the same time from different nodes, while being able to
access any resource, even when some nodes are down.

And in case of any partitions, that are much less likely to happen,
we'll just shut down all the lesser partitions, while only the largest
one will remain running, until the outage is resolved manually.

By master node, I mean the one that is used when we are joining new
nodes using riak-admin (as far as I remember, only one node can be
used for this). I believe, it's the one that is is returned by
riak_core_ring:owner_node/1.
Maybe, I'm wrong and when we call riak_core_gossip:send_ring(RingNode,
node()), we can use any node from the cluster
as RingNode?

Thank you.

On Thu, Jul 28, 2011 at 5:20 PM, Justin Sheehy <[hidden email]> wrote:

> Hi, Dmitry.
>
> A couple of suggestions...
>
> The reason that you're not seeing an easy way to automatically have nodes be added or removed from the cluster upon going down or coming up is that we recommend strongly against such behavior.
>
> The idea is that intentional (administrative) outages are very different in nature from unintentional and potentially transitory outages. We have explicit administrative commands such as "join" and "leave" for the administrative cases, making it very easy to add or remove hosts to a cluster. When a node is unreachable, you often can't automatically tell whether it is a host problem or a network problem and can't automatically tell if it is a long-term or short-term outage. This is why mechanisms such as quorums and hinted handoff exist: to ensure proper operation of the cluster as a whole throughout such outages. Consider the case where you have a network problem such that several of your nodes lose visibility to each other for brief and distinct periods of time. If nodes are auto-added and auto-removed then you will have quite a bit of churn and potentially a very harmful feedback scenario. Instead of auto-adding and auto-removing, consider using things like
> riak_core_node_watcher to decide which nodes to interact with on a per-operation basis.
>
> I'm also not sure what you mean by "if the master node goes down" since in most riak_core applications there is no master node. Of course you can create such a mechanism if you need it, but (e.g.) Riak KV and the accompanying applications do not have any notion of a master node and thus do not have any such concern.
>
> I hope that this is useful.
>
> Best regards,
>
> -Justin
>
>
>



--
Best regards,
Dmitry Demeshchuk

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riak_core questions

Justin Sheehy
Hi, Dmitry.

On Thu, Jul 28, 2011 at 12:22 PM, Dmitry Demeshchuk
<[hidden email]> wrote:

> By master node, I mean the one that is used when we are joining new
> nodes using riak-admin (as far as I remember, only one node can be
> used for this).

You can use any node at all in the existing cluster for this purpose.
They are all effectively identical.

-Justin

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com