Detecting Hinted Handoff

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Detecting Hinted Handoff

Hawk Newton
I've got a use case in which I'd like to use a secondary index but can't tolerate partial result sets caused by hinted handoffs.  I'm not currently running riak search and, as this is a fringe case, would prefer not add the additional overhead and complexity if I can help it.

I'd like to detect a hinted handoff operation and throw a 503, if possible.

Does anyone know of a way I can programatically detect if a hinted handoff is underway without having to shell out to riak-admin (yuck!) and parse the results? I'm running riak 2.0.5 at the moment.

Thank you in advance.

-- Hawk
Reply | Threaded
Open this post in threaded view
|

Re: Detecting Hinted Handoff

Alexander Sicular
Take a look at the "pw" and "pr" tunable consistency options for gets and puts. The base level of abstraction in Riak is the virtual node - not the physical machine. When data is replicated it is replicated to a replica set of virtual nodes. Those virtual nodes have primary and secondary (due to failures) allocations to physical machines. When using "pr" and "pw" options you instruct Riak to only service the request from virtual nodes that are residing on their primarily allocated physical machines. In short, by abusing pr/pw you can infer the state of your cluster from your application. 

Obviously, this is not foolproof. There may also be additional 2i specific issues to consider. Nevertheless, I always liked this trick. 

Also, review this four part series on tunable consistency :

-Alexander 


@siculars

Sent from my iRotaryPhone

On Jul 22, 2016, at 12:28, Hawk Newton <[hidden email]> wrote:

I've got a use case in which I'd like to use a secondary index but can't
tolerate partial result sets caused by hinted handoffs.  I'm not currently
running riak search and, as this is a fringe case, would prefer not add the
additional overhead and complexity if I can help it.

I'd like to detect a hinted handoff operation and throw a 503, if possible.

Does anyone know of a way I can programatically detect if a hinted handoff
is underway without having to shell out to riak-admin (yuck!) and parse the
results? I'm running riak 2.0.5 at the moment.

Thank you in advance.

-- Hawk



--
View this message in context: http://riak-users.197444.n3.nabble.com/Detecting-Hinted-Handoff-tp4034489.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Detecting Hinted Handoff

Sargun Dhillon
It might also make a lot of sense to roll your own secondary indices. That is, have a CRDT set represent the primary key of the rows which meet the 2i condition. In that, you can query the CRDT set, and ensure some level of consistency. There are further tricks to be played here if interested. 

I'm curious, what is your data model?

On Fri, Jul 22, 2016 at 11:01 AM, Alexander Sicular <[hidden email]> wrote:
Take a look at the "pw" and "pr" tunable consistency options for gets and puts. The base level of abstraction in Riak is the virtual node - not the physical machine. When data is replicated it is replicated to a replica set of virtual nodes. Those virtual nodes have primary and secondary (due to failures) allocations to physical machines. When using "pr" and "pw" options you instruct Riak to only service the request from virtual nodes that are residing on their primarily allocated physical machines. In short, by abusing pr/pw you can infer the state of your cluster from your application. 

Obviously, this is not foolproof. There may also be additional 2i specific issues to consider. Nevertheless, I always liked this trick. 

Also, review this four part series on tunable consistency :

-Alexander 


@siculars

Sent from my iRotaryPhone

On Jul 22, 2016, at 12:28, Hawk Newton <[hidden email]> wrote:

I've got a use case in which I'd like to use a secondary index but can't
tolerate partial result sets caused by hinted handoffs.  I'm not currently
running riak search and, as this is a fringe case, would prefer not add the
additional overhead and complexity if I can help it.

I'd like to detect a hinted handoff operation and throw a 503, if possible.

Does anyone know of a way I can programatically detect if a hinted handoff
is underway without having to shell out to riak-admin (yuck!) and parse the
results? I'm running riak 2.0.5 at the moment.

Thank you in advance.

-- Hawk



--
View this message in context: http://riak-users.197444.n3.nabble.com/Detecting-Hinted-Handoff-tp4034489.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Detecting Hinted Handoff

DeadZen
Hinted handoff from adding a node, removing a node or failing a node? Could probably get some idea from a ring handler, hinted handoff could likely very well trigger its own event as well without a large modification to riak_core 


On Sunday, July 24, 2016, Sargun Dhillon <[hidden email]> wrote:
It might also make a lot of sense to roll your own secondary indices. That is, have a CRDT set represent the primary key of the rows which meet the 2i condition. In that, you can query the CRDT set, and ensure some level of consistency. There are further tricks to be played here if interested. 

I'm curious, what is your data model?

On Fri, Jul 22, 2016 at 11:01 AM, Alexander Sicular <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;siculars@gmail.com&#39;);" target="_blank">siculars@...> wrote:
Take a look at the "pw" and "pr" tunable consistency options for gets and puts. The base level of abstraction in Riak is the virtual node - not the physical machine. When data is replicated it is replicated to a replica set of virtual nodes. Those virtual nodes have primary and secondary (due to failures) allocations to physical machines. When using "pr" and "pw" options you instruct Riak to only service the request from virtual nodes that are residing on their primarily allocated physical machines. In short, by abusing pr/pw you can infer the state of your cluster from your application. 

Obviously, this is not foolproof. There may also be additional 2i specific issues to consider. Nevertheless, I always liked this trick. 

Also, review this four part series on tunable consistency :

-Alexander 


@siculars

Sent from my iRotaryPhone

On Jul 22, 2016, at 12:28, Hawk Newton <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;hawk.newton@gmail.com&#39;);" target="_blank">hawk.newton@...> wrote:

I've got a use case in which I'd like to use a secondary index but can't
tolerate partial result sets caused by hinted handoffs.  I'm not currently
running riak search and, as this is a fringe case, would prefer not add the
additional overhead and complexity if I can help it.

I'd like to detect a hinted handoff operation and throw a 503, if possible.

Does anyone know of a way I can programatically detect if a hinted handoff
is underway without having to shell out to riak-admin (yuck!) and parse the
results? I'm running riak 2.0.5 at the moment.

Thank you in advance.

-- Hawk



--
View this message in context: http://riak-users.197444.n3.nabble.com/Detecting-Hinted-Handoff-tp4034489.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak-users@lists.basho.com&#39;);" target="_blank">riak-users@...
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak-users@lists.basho.com&#39;);" target="_blank">riak-users@...
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Detecting Hinted Handoff

Russell Brown-2
But I still don’t understand _why_? What is the aim, ‘cos I’m not sure it’s possible.

For quorum reads on indexes then Sargun’s “roll your own” idea is the better answer at the moment.

On 25 Jul 2016, at 11:25, DeadZen <[hidden email]> wrote:

> Hinted handoff from adding a node, removing a node or failing a node? Could probably get some idea from a ring handler, hinted handoff could likely very well trigger its own event as well without a large modification to riak_core
>
>
> On Sunday, July 24, 2016, Sargun Dhillon <[hidden email]> wrote:
> It might also make a lot of sense to roll your own secondary indices. That is, have a CRDT set represent the primary key of the rows which meet the 2i condition. In that, you can query the CRDT set, and ensure some level of consistency. There are further tricks to be played here if interested.
>
> I'm curious, what is your data model?
>
> On Fri, Jul 22, 2016 at 11:01 AM, Alexander Sicular <[hidden email]> wrote:
> Take a look at the "pw" and "pr" tunable consistency options for gets and puts. The base level of abstraction in Riak is the virtual node - not the physical machine. When data is replicated it is replicated to a replica set of virtual nodes. Those virtual nodes have primary and secondary (due to failures) allocations to physical machines. When using "pr" and "pw" options you instruct Riak to only service the request from virtual nodes that are residing on their primarily allocated physical machines. In short, by abusing pr/pw you can infer the state of your cluster from your application.
>
> Obviously, this is not foolproof. There may also be additional 2i specific issues to consider. Nevertheless, I always liked this trick.
>
> Also, review this four part series on tunable consistency :
>
> http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/
> http://basho.com/posts/technical/riaks-config-behaviors-part-2/
> http://basho.com/posts/technical/riaks-config-behaviors-part-3/
> http://basho.com/posts/technical/riaks-config-behaviors-part-4/
>
> -Alexander
>
>
> @siculars
> http://siculars.posthaven.com
>
> Sent from my iRotaryPhone
>
> On Jul 22, 2016, at 12:28, Hawk Newton <[hidden email]> wrote:
>
>> I've got a use case in which I'd like to use a secondary index but can't
>> tolerate partial result sets caused by hinted handoffs.  I'm not currently
>> running riak search and, as this is a fringe case, would prefer not add the
>> additional overhead and complexity if I can help it.
>>
>> I'd like to detect a hinted handoff operation and throw a 503, if possible.
>>
>> Does anyone know of a way I can programatically detect if a hinted handoff
>> is underway without having to shell out to riak-admin (yuck!) and parse the
>> results? I'm running riak 2.0.5 at the moment.
>>
>> Thank you in advance.
>>
>> -- Hawk
>>
>>
>>
>> --
>> View this message in context: http://riak-users.197444.n3.nabble.com/Detecting-Hinted-Handoff-tp4034489.html
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Detecting Hinted Handoff

DeadZen
Agreed rolling your own 2i is likely your best bet here, I too am wondering what purpose it would serve you. 


On Monday, July 25, 2016, Russell Brown <[hidden email]> wrote:
But I still don’t understand _why_? What is the aim, ‘cos I’m not sure it’s possible.

For quorum reads on indexes then Sargun’s “roll your own” idea is the better answer at the moment.

On 25 Jul 2016, at 11:25, DeadZen <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;deadzen@deadzen.com&#39;)">deadzen@...> wrote:

> Hinted handoff from adding a node, removing a node or failing a node? Could probably get some idea from a ring handler, hinted handoff could likely very well trigger its own event as well without a large modification to riak_core
>
>
> On Sunday, July 24, 2016, Sargun Dhillon <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;sargun@sargun.me&#39;)">sargun@...> wrote:
> It might also make a lot of sense to roll your own secondary indices. That is, have a CRDT set represent the primary key of the rows which meet the 2i condition. In that, you can query the CRDT set, and ensure some level of consistency. There are further tricks to be played here if interested.
>
> I'm curious, what is your data model?
>
> On Fri, Jul 22, 2016 at 11:01 AM, Alexander Sicular <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;siculars@gmail.com&#39;)">siculars@...> wrote:
> Take a look at the "pw" and "pr" tunable consistency options for gets and puts. The base level of abstraction in Riak is the virtual node - not the physical machine. When data is replicated it is replicated to a replica set of virtual nodes. Those virtual nodes have primary and secondary (due to failures) allocations to physical machines. When using "pr" and "pw" options you instruct Riak to only service the request from virtual nodes that are residing on their primarily allocated physical machines. In short, by abusing pr/pw you can infer the state of your cluster from your application.
>
> Obviously, this is not foolproof. There may also be additional 2i specific issues to consider. Nevertheless, I always liked this trick.
>
> Also, review this four part series on tunable consistency :
>
> http://basho.com/posts/technical/understanding-riaks-configurable-behaviors-part-1/
> http://basho.com/posts/technical/riaks-config-behaviors-part-2/
> http://basho.com/posts/technical/riaks-config-behaviors-part-3/
> http://basho.com/posts/technical/riaks-config-behaviors-part-4/
>
> -Alexander
>
>
> @siculars
> http://siculars.posthaven.com
>
> Sent from my iRotaryPhone
>
> On Jul 22, 2016, at 12:28, Hawk Newton <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;hawk.newton@gmail.com&#39;)">hawk.newton@...> wrote:
>
>> I've got a use case in which I'd like to use a secondary index but can't
>> tolerate partial result sets caused by hinted handoffs.  I'm not currently
>> running riak search and, as this is a fringe case, would prefer not add the
>> additional overhead and complexity if I can help it.
>>
>> I'd like to detect a hinted handoff operation and throw a 503, if possible.
>>
>> Does anyone know of a way I can programatically detect if a hinted handoff
>> is underway without having to shell out to riak-admin (yuck!) and parse the
>> results? I'm running riak 2.0.5 at the moment.
>>
>> Thank you in advance.
>>
>> -- Hawk
>>
>>
>>
>> --
>> View this message in context: http://riak-users.197444.n3.nabble.com/Detecting-Hinted-Handoff-tp4034489.html
>> Sent from the Riak Users mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> riak-users mailing list
>> <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> _______________________________________________
> riak-users mailing list
> <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com