Stale index entries after delete

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Stale index entries after delete

Bernard Duggan
Hi list,
We're seeing some weird behaviour that I haven't been able to find any decent explanation for. Essentially, after deleting records out of a bucket, the records continue to persist in the solr index for that bucket type.

Things that might matter:
* We're using v2.2.1 (and also saw it on 2.2.0)
* We're using the `memory` storage backend for Riak.
* The indices showing the problem have various schemas, but it happens with `_yz_default` as easily as any others.
* `anti_entropy` and `search` are both turned on.
* This happens on single-node developer "clusters" with n_val set to 3. We haven't tried any other configurations yet.

The problem doesn't seem to take any special actions to trigger - we write a few objects to a bucket and they show up in the index. Then we delete them, wait a few seconds to make sure everything's caught up, and read the index and they're often (though it seems not always - I haven't been able to pin down the exact circumstances) still there. It's also reproducible if we stop and start riak (which clears out the memory-backed buckets but doesn't seem to do the same for the indices).

Is this a known issue with/feature of memory-backed buckets or is there somewhere else we should look?

Cheers,

Bernard

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Stale index entries after delete

Fred Dushin
Hello Bernard,

Thank you for writing this up, and apologies for the delay in responding.

I have not been able to readily reproduce this in my test environment, and our Riak tests (e.g., [1]) test scenarios very much like this.

If this is a test environment, can you try stopping riak, deleting your data/yz_anti_entropy directory, and restarting?  Monitor the output of riak-admin search aae-status for all trees to rebuild and all exchanges to complete, and check to verify that your search results are correct?

Also, if you have any additional bucket properties configured (e.g., strong consistency, allow_mult, lww, etc), please send those along.

[1] https://github.com/basho/yokozuna/blob/develop/riak_test/yz_siblings.erl

> On Mar 14, 2017, at 11:39 PM, Bernard Duggan <[hidden email]> wrote:
>
> Hi list,
> We're seeing some weird behaviour that I haven't been able to find any decent explanation for. Essentially, after deleting records out of a bucket, the records continue to persist in the solr index for that bucket type.
>
> Things that might matter:
> * We're using v2.2.1 (and also saw it on 2.2.0)
> * We're using the `memory` storage backend for Riak.
> * The indices showing the problem have various schemas, but it happens with `_yz_default` as easily as any others.
> * `anti_entropy` and `search` are both turned on.
> * This happens on single-node developer "clusters" with n_val set to 3. We haven't tried any other configurations yet.
>
> The problem doesn't seem to take any special actions to trigger - we write a few objects to a bucket and they show up in the index. Then we delete them, wait a few seconds to make sure everything's caught up, and read the index and they're often (though it seems not always - I haven't been able to pin down the exact circumstances) still there. It's also reproducible if we stop and start riak (which clears out the memory-backed buckets but doesn't seem to do the same for the indices).
>
> Is this a known issue with/feature of memory-backed buckets or is there somewhere else we should look?
>
> Cheers,
>
> Bernard
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...