N = 3 and RW = 2 not finding some keys

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

N = 3 and RW = 2 not finding some keys

Guido Medina-3
Hi all,

After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: N = 3 and RW = 2 not finding some keys

Bryan Hunt-3
Hi Guido, 

You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.

I.e 

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.


From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

On 17 May 2018, at 15:42, Guido Medina <[hidden email]> wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: N = 3 and RW = 2 not finding some keys

Bryan Hunt-3
Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate. 

On 18 May 2018, at 11:00, Bryan Hunt <[hidden email]> wrote:

Hi Guido, 

You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.

I.e 

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.


From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

On 17 May 2018, at 15:42, Guido Medina <[hidden email]> wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: N = 3 and RW = 2 not finding some keys

Russell Brown-4
But why isn’t read repair “working”?

> On 18 May 2018, at 11:07, Bryan Hunt <[hidden email]> wrote:
>
> Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate.
>
>> On 18 May 2018, at 11:00, Bryan Hunt <[hidden email]> wrote:
>>
>> Hi Guido,
>>
>> You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.
>>
>> I.e
>>
>> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'
>>
>> This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.
>>
>> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>>
>> From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.
>>
>> Best Regards,
>>
>> Bryan Hunt
>>
>>> On 17 May 2018, at 15:42, Guido Medina <[hidden email]> wrote:
>>>
>>> Hi all,
>>>
>>> After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2
>>>
>>> Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.
>>>
>>> Any help will be truly appreciated.
>>>
>>> Kind regards,
>>> Guido.
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: N = 3 and RW = 2 not finding some keys

Guido Medina-3
A force replace done wrong causing some partitions to only have one copy.
And the cluster running extremely slow have forced us to disable AAE for now.

Guido.

On 18/05/18 13:13, Russell Brown wrote:
But why isn’t read repair “working”?

On 18 May 2018, at 11:07, Bryan Hunt [hidden email] wrote:

Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate. 

On 18 May 2018, at 11:00, Bryan Hunt [hidden email] wrote:

Hi Guido, 

You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.

I.e 

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

On 17 May 2018, at 15:42, Guido Medina [hidden email] wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

        
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: N = 3 and RW = 2 not finding some keys

Bryan Hunt-3
In reply to this post by Russell Brown-4
Russell, Good question. I’m guessing they are iterating, and requesting a different object for each request?

Guido, given the behaviour you initially described, before applying the configuration I suggested - did you receive a successful response upon subsequent requests for the same object ?

> On 18 May 2018, at 13:13, Russell Brown <[hidden email]> wrote:
>
> But why isn’t read repair “working”?
>
>> On 18 May 2018, at 11:07, Bryan Hunt <[hidden email]> wrote:
>>
>> Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate.
>>
>>> On 18 May 2018, at 11:00, Bryan Hunt <[hidden email]> wrote:
>>>
>>> Hi Guido,
>>>
>>> You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.
>>>
>>> I.e
>>>
>>> curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'
>>>
>>> This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.
>>>
>>> https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
>>>
>>> From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.
>>>
>>> Best Regards,
>>>
>>> Bryan Hunt
>>>
>>>> On 17 May 2018, at 15:42, Guido Medina <[hidden email]> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2
>>>>
>>>> Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.
>>>>
>>>> Any help will be truly appreciated.
>>>>
>>>> Kind regards,
>>>> Guido.
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> [hidden email]
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: N = 3 and RW = 2 not finding some keys

Guido Medina-3
Yes, but we haven't set back R = 2, we are currently with R = 3 until we resolve the issue.
I'm in the middle of writing something that will read every single key for the cluster using our relation DB IDs (each key has a counterpart on DB)

I'm guessing that 2i streaming won't help? we need to make sure we are fully consistent (for RW = 2) so I have to find a way to repair everything before turning R to 2.

After reading the object yes every request after was successful, but we even had a weird situation where after writing key with W = 2 we couldn't find with R = 2, not good for us.

Guido.

On 18/05/18 14:34, Bryan Hunt wrote:
Russell, Good question. I’m guessing they are iterating, and requesting a different object for each request? 

Guido, given the behaviour you initially described, before applying the configuration I suggested - did you receive a successful response upon subsequent requests for the same object ?

On 18 May 2018, at 13:13, Russell Brown [hidden email] wrote:

But why isn’t read repair “working”?

On 18 May 2018, at 11:07, Bryan Hunt [hidden email] wrote:

Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate. 

On 18 May 2018, at 11:00, Bryan Hunt [hidden email] wrote:

Hi Guido, 

You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.

I.e 

curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'

This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.

https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok

From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.

Best Regards,

Bryan Hunt

On 17 May 2018, at 15:42, Guido Medina [hidden email] wrote:

Hi all,

After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2

Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.

Any help will be truly appreciated.

Kind regards,
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

          
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

      

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

RE: N = 3 and RW = 2 not finding some keys

Nicholas Adams

Sorry for being late to the party but have you tried repairing all partitions? https://docs.basho.com/riak/kv/2.1.4/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

 

This forces every single piece of information to be read and thus causes read repair to kick in for any partitions that register less than 3 copies.

 

Best regards,

 

Nicholas

 

From: riak-users <[hidden email]> On Behalf Of Guido Medina
Sent: 18 May 2018 22:52
To: riak-users <[hidden email]>
Subject: Re: N = 3 and RW = 2 not finding some keys

 

Yes, but we haven't set back R = 2, we are currently with R = 3 until we resolve the issue.
I'm in the middle of writing something that will read every single key for the cluster using our relation DB IDs (each key has a counterpart on DB)

I'm guessing that 2i streaming won't help? we need to make sure we are fully consistent (for RW = 2) so I have to find a way to repair everything before turning R to 2.

After reading the object yes every request after was successful, but we even had a weird situation where after writing key with W = 2 we couldn't find with R = 2, not good for us.

Guido.

On 18/05/18 14:34, Bryan Hunt wrote:

Russell, Good question. I’m guessing they are iterating, and requesting a different object for each request? 
 
Guido, given the behaviour you initially described, before applying the configuration I suggested - did you receive a successful response upon subsequent requests for the same object ?
 
On 18 May 2018, at 13:13, Russell Brown [hidden email] wrote:
 
But why isn’t read repair “working”?
 
On 18 May 2018, at 11:07, Bryan Hunt [hidden email] wrote:
 
Of course, AAE will eventually repair the missing object replicas but it seems like you need something more immediate. 
 
On 18 May 2018, at 11:00, Bryan Hunt [hidden email] wrote:
 
Hi Guido, 
 
You should attempt to change the bucket property ‘notfound_ok’ from the default of ‘true' to ‘false'.
 
I.e 
 
curl -XPUT 127.0.0.1:10018/buckets/foo/props -H "Content-Type: application/json" -d '{"props":{"notfound_ok": false}}'
 
This makes GET operations for non-existent keys slower as it forces an internal GET for each of the three copies.
 
https://docs.basho.com/riak/kv/2.1.1/developing/app-guide/replication-properties/#the-implications-of-notfound-ok
 
From what you describe, it sounds like only a single copy (out of the original three), somehow remain present in your cluster.
 
Best Regards,
 
Bryan Hunt
 
On 17 May 2018, at 15:42, Guido Medina [hidden email] wrote:
 
Hi all,
 
After some big rebalance of our cluster some keys are not found anymore unless we set R = 3, we had N = 3 and R = W = 2
 
Is there any sort of repair that would correct such situation for Riak 2.2.3, this is really driving us nuts.
 
Any help will be truly appreciated.
 
Kind regards,
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 
 
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
 
 
 
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

 


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Partitions repair isn't working anymore?

Guido Medina-3
Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
([hidden email])3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:
{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, 'riak@node4.domain.com'} <- riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Russell Brown-4
riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell

> On 1 Jun 2018, at 09:07, Guido Medina <[hidden email]> wrote:
>
> Hi all,
>
> We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
>> ([hidden email])3> [riak_search_vnode:repair(P) || P <- Partitions].
>> ** exception error: undefined function riak_search_vnode:repair/1
>
> The first two steps for the node repair executed fine:
>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>> Partitions = [P || {P, '[hidden email]'} <- riak_core_ring:all_owners(Ring)].
>
> We are running on 2.2.5
> Guido.
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

RE: Partitions repair isn't working anymore?

Nicholas Adams
From the mail history, I'd guess he's following the standard repair process listed on https://docs.basho.com/riak/kv/2.1.4/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node


-----Original Message-----
From: riak-users <[hidden email]> On Behalf Of Russell Brown
Sent: 01 June 2018 17:16
To: Guido Medina <[hidden email]>
Cc: riak-users <[hidden email]>
Subject: Re: Partitions repair isn't working anymore?

riak_search has been removed from riak-2.2.5. Looks like some vestige survived.

Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell

> On 1 Jun 2018, at 09:07, Guido Medina <[hidden email]> wrote:
>
> Hi all,
>
> We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
>> ([hidden email])3> [riak_search_vnode:repair(P) || P <- Partitions].
>> ** exception error: undefined function riak_search_vnode:repair/1
>
> The first two steps for the node repair executed fine:
>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>> Partitions = [P || {P, '[hidden email]'} <- riak_core_ring:all_owners(Ring)].
>
> We are running on 2.2.5
> Guido.
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Guido Medina-3
In reply to this post by Russell Brown-4
Hi Russell,

I was repairing each node as specified in this guide https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:
riak_search has been removed from riak-2.2.5. Looks like some vestige survived. 

Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell

On 1 Jun 2018, at 09:07, Guido Medina [hidden email] wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
([hidden email])3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1
The first two steps for the node repair executed fine:
{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '[hidden email]'} <- riak_core_ring:all_owners(Ring)].
We are running on 2.2.5
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Guido Medina-3
Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:
Hi Russell,

I was repairing each node as specified in this guide https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:
riak_search has been removed from riak-2.2.5. Looks like some vestige survived. 

Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell

On 1 Jun 2018, at 09:07, Guido Medina [hidden email] wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
([hidden email])3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1
The first two steps for the node repair executed fine:
{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '[hidden email]'} <- riak_core_ring:all_owners(Ring)].
We are running on 2.2.5
Guido.
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

      



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Russell Brown-4
I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?

> On 1 Jun 2018, at 09:35, Guido Medina <[hidden email]> wrote:
>
> Sorry, not repairing a single partition but all partitions per node:
> https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node
>
> On 01/06/18 09:34, Guido Medina wrote:
>> Hi Russell,
>>
>> I was repairing each node as specified in this guide https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition
>>
>> Guido.
>>
>> On 01/06/18 09:16, Russell Brown wrote:
>>> riak_search has been removed from riak-2.2.5. Looks like some vestige survived.
>>>
>>> Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?
>>>
>>> Is this some snippet that you attach and run?
>>>
>>> Cheers
>>>
>>> Russell
>>>
>>>
>>>> On 1 Jun 2018, at 09:07, Guido Medina <[hidden email]>
>>>>  wrote:
>>>>
>>>> Hi all,
>>>>
>>>> We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
>>>>
>>>>> ([hidden email]
>>>>> )3> [riak_search_vnode:repair(P) || P <- Partitions].
>>>>> ** exception error: undefined function riak_search_vnode:repair/1
>>>>>
>>>> The first two steps for the node repair executed fine:
>>>>
>>>>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>>>>> Partitions = [P || {P, '
>>>>> [hidden email]
>>>>> '} <- riak_core_ring:all_owners(Ring)].
>>>>>
>>>> We are running on 2.2.5
>>>> Guido.
>>>> _______________________________________________
>>>> riak-users mailing list
>>>>
>>>> [hidden email]
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Guido Medina-3
What did I copied and pasted, sorry about that, it must be that it is Friday or something...my apologies.

On 01/06/18 09:37, Russell Brown wrote:
I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?

On 1 Jun 2018, at 09:35, Guido Medina [hidden email] wrote:

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:
Hi Russell,

I was repairing each node as specified in this guide https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:
riak_search has been removed from riak-2.2.5. Looks like some vestige survived. 

Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell


On 1 Jun 2018, at 09:07, Guido Medina [hidden email]
 wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:

([hidden email]
)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '
[hidden email]
'} <- riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.
_______________________________________________
riak-users mailing list

[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

        
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Guido Medina-3
In reply to this post by Russell Brown-4
The last command should have been like the following:
  • {ok, Ring} = riak_core_ring_manager:get_my_ring().
  • Partitions = [P || {P, '[hidden email]'} <- riak_core_ring:all_owners(Ring)].
  • [riak_kv_vnode:repair(P) || P <- Partitions].
My bad, I think I copied the command from the wrong instructions before.

Guido.

On 01/06/18 09:37, Russell Brown wrote:
I don’t see a call to `riak_search_vnode:repair` in those docs

Do you still run legacy riak search (i.e. not yokozuna/solr)?

On 1 Jun 2018, at 09:35, Guido Medina [hidden email] wrote:

Sorry, not repairing a single partition but all partitions per node:
https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node

On 01/06/18 09:34, Guido Medina wrote:
Hi Russell,

I was repairing each node as specified in this guide https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition

Guido.

On 01/06/18 09:16, Russell Brown wrote:
riak_search has been removed from riak-2.2.5. Looks like some vestige survived. 

Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?

Is this some snippet that you attach and run?

Cheers

Russell


On 1 Jun 2018, at 09:07, Guido Medina [hidden email]
 wrote:

Hi all,

We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:

([hidden email]
)3> [riak_search_vnode:repair(P) || P <- Partitions].
** exception error: undefined function riak_search_vnode:repair/1

The first two steps for the node repair executed fine:

{ok, Ring} = riak_core_ring_manager:get_my_ring().
Partitions = [P || {P, '
[hidden email]
'} <- riak_core_ring:all_owners(Ring)].

We are running on 2.2.5
Guido.
_______________________________________________
riak-users mailing list

[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

        
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Partitions repair isn't working anymore?

Russell Brown-4

> On 1 Jun 2018, at 10:07, Guido Medina <[hidden email]> wrote:
>
> The last command should have been like the following:
> • {ok, Ring} = riak_core_ring_manager:get_my_ring().
> • Partitions = [P || {P, '[hidden email]'} <- riak_core_ring:all_owners(Ring)].
> • [riak_kv_vnode:repair(P) || P <- Partitions].
> My bad, I think I copied the command from the wrong instructions before.

And does that work?

From your mail you said that
>>>>>>> )3> [riak_search_vnode:repair(P) || P <- Partitions].
>>>>>>> ** exception error: undefined function riak_search_vnode:repair/1
>>>>>>>

Does not work. This I would expect, and I was asking, why do you run this? Are you using legacy search?

If you run

> • [riak_kv_vnode:repair(P) || P <- Partitions].

Does it work?

Cheers

Russell

>
> Guido.
>
> On 01/06/18 09:37, Russell Brown wrote:
>> I don’t see a call to `riak_search_vnode:repair` in those docs
>>
>> Do you still run legacy riak search (i.e. not yokozuna/solr)?
>>
>>
>>> On 1 Jun 2018, at 09:35, Guido Medina <[hidden email]>
>>>  wrote:
>>>
>>> Sorry, not repairing a single partition but all partitions per node:
>>>
>>> https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-all-partitions-on-a-node
>>>
>>>
>>> On 01/06/18 09:34, Guido Medina wrote:
>>>
>>>> Hi Russell,
>>>>
>>>> I was repairing each node as specified in this guide
>>>> https://docs.basho.com/riak/kv/2.2.3/using/repair-recovery/repairs/#repairing-a-single-partition
>>>>
>>>>
>>>> Guido.
>>>>
>>>> On 01/06/18 09:16, Russell Brown wrote:
>>>>
>>>>> riak_search has been removed from riak-2.2.5. Looks like some vestige survived.
>>>>>
>>>>> Can you tell me what command you ran, it looks to me from the output below that you’re connected to node and typing commands in the console?
>>>>>
>>>>> Is this some snippet that you attach and run?
>>>>>
>>>>> Cheers
>>>>>
>>>>> Russell
>>>>>
>>>>>
>>>>>
>>>>>> On 1 Jun 2018, at 09:07, Guido Medina <[hidden email]>
>>>>>>
>>>>>>  wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We started the partitions repair a couple of weeks ago, so far so good (3 nodes out of 7 done), then we started getting this error:
>>>>>>
>>>>>>
>>>>>>> ([hidden email]
>>>>>>>
>>>>>>> )3> [riak_search_vnode:repair(P) || P <- Partitions].
>>>>>>> ** exception error: undefined function riak_search_vnode:repair/1
>>>>>>>
>>>>>>>
>>>>>> The first two steps for the node repair executed fine:
>>>>>>
>>>>>>
>>>>>>> {ok, Ring} = riak_core_ring_manager:get_my_ring().
>>>>>>> Partitions = [P || {P, '
>>>>>>>
>>>>>>> [hidden email]
>>>>>>>
>>>>>>> '} <- riak_core_ring:all_owners(Ring)].
>>>>>>>
>>>>>>>
>>>>>> We are running on 2.2.5
>>>>>> Guido.
>>>>>> _______________________________________________
>>>>>> riak-users mailing list
>>>>>>
>>>>>>
>>>>>> [hidden email]
>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>> _______________________________________________
>>> riak-users mailing list
>>>
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com