Reaping Tombstones

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Reaping Tombstones

Arun Rajagopalan
Hello Riakers

I wonder if there is anyway to delete all tombstones that were left behind when delete_mode was set to 'keep'

Also how do I estimate how much space the tombstones take ? We have a ton of large scale and frequent delete operations and I suspect some space is used by Tombstones. I would like to find out if it is significant enough to warrant some cleanup. If it does, how do I remove them?

Thanks and wish you all Happy, Peaceful and Prosperous  2017 !



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Reaping Tombstones

Matthew Von-Maszewski
Greetings,

I am not able to answer your tombstone questions.  That question needs a better expert.

Just wanted to point out that Riak now has global expiry in both the leveldb and bitcask backends.  That might be a quicker solution for your frequent delete operations:


Technical details for the leveldb expiry are found here:
  https://github.com/basho/leveldb/wiki/mv-expiry

Matthew

On Dec 30, 2016, at 9:55 AM, Arun Rajagopalan <[hidden email]> wrote:

Hello Riakers

I wonder if there is anyway to delete all tombstones that were left behind when delete_mode was set to 'keep'

Also how do I estimate how much space the tombstones take ? We have a ton of large scale and frequent delete operations and I suspect some space is used by Tombstones. I would like to find out if it is significant enough to warrant some cleanup. If it does, how do I remove them?

Thanks and wish you all Happy, Peaceful and Prosperous  2017 !


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Reaping Tombstones

Luca Favatella
On 30 December 2016 at 15:06, Matthew Von-Maszewski <[hidden email]> wrote:

> Greetings,
>
> I am not able to answer your tombstone questions.  That question needs a
> better expert.
>
> Just wanted to point out that Riak now has global expiry in both the leveldb
> and bitcask backends.  That might be a quicker solution for your frequent
> delete operations:
>
>   http://basho.com/products/riak-kv/global-object-expiration/
>
> Technical details for the leveldb expiry are found here:
>   https://github.com/basho/leveldb/wiki/mv-expiry
>
> Matthew
>
> On Dec 30, 2016, at 9:55 AM, Arun Rajagopalan <[hidden email]>
> wrote:
>
> Hello Riakers
>
> I wonder if there is anyway to delete all tombstones that were left behind
> when delete_mode was set to 'keep'

Hi Arun,

Brute-force rate-limited map-reduce custom application code looking
for `X-Riak-Deleted` in object metadata and then deleting? Refs:
* http://docs.basho.com/riak/kv/2.2.0/developing/app-guide/advanced-mapreduce/#map-phase
* https://github.com/basho/basho_docs/blob/6b2f42a7243bfded737bd62eed12c490376c67e2/content/riak/kv/2.2.0/developing/app-guide/advanced-mapreduce.md#map-phase

Interested to hear if you come up with a better way though.


> Also how do I estimate how much space the tombstones take ? We have a ton of
> large scale and frequent delete operations and I suspect some space is used
> by Tombstones. I would like to find out if it is significant enough to
> warrant some cleanup.

I understand this is a separate question - I am not sure about the answer.


> If it does, how do I remove them?

I believe I proposed an answer above.


Regards
Luca

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Reaping Tombstones

Arun Rajagopalan
Thanks Matthew & Luca

Re: global expiry - will that option retroactively remove objects? That is remove objects that became "unneeded" before the option was set ?
Same question w.r.t delete_mode

Re: Map / Reduce - I am not sure the delete would remove the tombstone unless I set the delete_mode to immediate AND there are no copies on non-primary nodes. Or am I mistaken ?


On Fri, Dec 30, 2016 at 10:11 AM, Luca Favatella <[hidden email]> wrote:
On 30 December 2016 at 15:06, Matthew Von-Maszewski <[hidden email]> wrote:
> Greetings,
>
> I am not able to answer your tombstone questions.  That question needs a
> better expert.
>
> Just wanted to point out that Riak now has global expiry in both the leveldb
> and bitcask backends.  That might be a quicker solution for your frequent
> delete operations:
>
>   http://basho.com/products/riak-kv/global-object-expiration/
>
> Technical details for the leveldb expiry are found here:
>   https://github.com/basho/leveldb/wiki/mv-expiry
>
> Matthew
>
> On Dec 30, 2016, at 9:55 AM, Arun Rajagopalan <[hidden email]>
> wrote:
>
> Hello Riakers
>
> I wonder if there is anyway to delete all tombstones that were left behind
> when delete_mode was set to 'keep'

Hi Arun,

Brute-force rate-limited map-reduce custom application code looking
for `X-Riak-Deleted` in object metadata and then deleting? Refs:
* http://docs.basho.com/riak/kv/2.2.0/developing/app-guide/advanced-mapreduce/#map-phase
* https://github.com/basho/basho_docs/blob/6b2f42a7243bfded737bd62eed12c490376c67e2/content/riak/kv/2.2.0/developing/app-guide/advanced-mapreduce.md#map-phase

Interested to hear if you come up with a better way though.


> Also how do I estimate how much space the tombstones take ? We have a ton of
> large scale and frequent delete operations and I suspect some space is used
> by Tombstones. I would like to find out if it is significant enough to
> warrant some cleanup.

I understand this is a separate question - I am not sure about the answer.


> If it does, how do I remove them?

I believe I proposed an answer above.


Regards
Luca


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Reaping Tombstones

Matthew Von-Maszewski
The current release of global expiry does not support retroactively dating and removing objects.  Only newly written objects get expiry.

The code for retroactively dating objects exists and will be part of a future release.

Matthew


On Dec 30, 2016, at 10:30 AM, Arun Rajagopalan <[hidden email]> wrote:

Thanks Matthew & Luca

Re: global expiry - will that option retroactively remove objects? That is remove objects that became "unneeded" before the option was set ?
Same question w.r.t delete_mode

Re: Map / Reduce - I am not sure the delete would remove the tombstone unless I set the delete_mode to immediate AND there are no copies on non-primary nodes. Or am I mistaken ?


On Fri, Dec 30, 2016 at 10:11 AM, Luca Favatella <[hidden email]> wrote:
On 30 December 2016 at 15:06, Matthew Von-Maszewski <[hidden email]> wrote:
> Greetings,
>
> I am not able to answer your tombstone questions.  That question needs a
> better expert.
>
> Just wanted to point out that Riak now has global expiry in both the leveldb
> and bitcask backends.  That might be a quicker solution for your frequent
> delete operations:
>
>   http://basho.com/products/riak-kv/global-object-expiration/
>
> Technical details for the leveldb expiry are found here:
>   https://github.com/basho/leveldb/wiki/mv-expiry
>
> Matthew
>
> On Dec 30, 2016, at 9:55 AM, Arun Rajagopalan <[hidden email]>
> wrote:
>
> Hello Riakers
>
> I wonder if there is anyway to delete all tombstones that were left behind
> when delete_mode was set to 'keep'

Hi Arun,

Brute-force rate-limited map-reduce custom application code looking
for `X-Riak-Deleted` in object metadata and then deleting? Refs:
* http://docs.basho.com/riak/kv/2.2.0/developing/app-guide/advanced-mapreduce/#map-phase
* https://github.com/basho/basho_docs/blob/6b2f42a7243bfded737bd62eed12c490376c67e2/content/riak/kv/2.2.0/developing/app-guide/advanced-mapreduce.md#map-phase

Interested to hear if you come up with a better way though.


> Also how do I estimate how much space the tombstones take ? We have a ton of
> large scale and frequent delete operations and I suspect some space is used
> by Tombstones. I would like to find out if it is significant enough to
> warrant some cleanup.

I understand this is a separate question - I am not sure about the answer.


> If it does, how do I remove them?

I believe I proposed an answer above.


Regards
Luca



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Reaping Tombstones

Nick Marino
In reply to this post by Arun Rajagopalan
We have new tombstone reaping functionality in the works for Riak KV 2.3, which should allow for safe and automatic removal of old leftover tombstones. In the mean time, you can potentially trigger the deletion of old, leftover tombstones by doing reads to those keys; if all the primary replicas return the tombstone then Riak will automatically try and remove it.

Note, however, that this will not immediately reclaim disk space. Deletion of the tombstone by Riak will still leave behind a backend tombstone for some amount of time. See the docs here for more info on this: http://docs.basho.com/riak/kv/2.2.0/using/reference/object-deletion/#deletion-from-backends

As for estimating disk space used by tombstones, I'm afraid I don't know of any easy way to do that. Hopefully someone else may have a good suggestion here.

Cheers,
Nick

On Fri, Dec 30, 2016 at 9:55 AM, Arun Rajagopalan <[hidden email]> wrote:
Hello Riakers

I wonder if there is anyway to delete all tombstones that were left behind when delete_mode was set to 'keep'

Also how do I estimate how much space the tombstones take ? We have a ton of large scale and frequent delete operations and I suspect some space is used by Tombstones. I would like to find out if it is significant enough to warrant some cleanup. If it does, how do I remove them?

Thanks and wish you all Happy, Peaceful and Prosperous  2017 !



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com