Riak: reliable object deletion

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Riak: reliable object deletion

al so
 Here is brief env:
 Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
 
 Problem:
  Deleted object suddenly resurrected after few days. Solr search query("*:*") was executed around the time of reappearance.

 Bucket property for this reappeared object
 {
  "props": {
    "name": "UsaHype",
    "allow_mult": false,
    "basic_quorum": false,
    "big_vclock": 50,
    "chash_keyfun": {
      "mod": "riak_core_util",
      "fun": "chash_std_keyfun"
    },
    "dvv_enabled": true,
    "dw": "quorum",
    "last_write_wins": false,
    "linkfun": {
      "mod": "riak_kv_wm_link_walker",
      "fun": "mapreduce_linkfun"
    },
    "n_val": 3,
    "notfound_ok": true,
    "old_vclock": 86400,
    "postcommit": [
      {
        "mod": "riak_repl_leader",
        "fun": "postcommit"
      },
      {
        "mod": "riak_repl2_rt",
        "fun": "postcommit"
      }
    ],
    "pr": 0,
    "precommit": [
      
    ],
    "pw": 0,
    "r": "quorum",
    "repl": true,
    "rw": "quorum",
    "small_vclock": 50,
    "w": "quorum",
    "young_vclock": 20
  }
}
 
 delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still get created when no delete_mode at all in the config? Can we query metadata(X-Riak-Deleted) for this reappeared object.
 How would one go about finding root cause. I do know the date when the object was deleted from our app logs. I also seem to know when it reappeared. Seems like a solr search ("*:*") in code was executed just before this object reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are out-of-sync (AAE?). 

 I understand tuning other params (allow_mult = true, ...) will have its own implications.

 In summary:
  How do I find root cause of this issue. 
  How to reliable delete an object.

-Volk


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak: reliable object deletion

Fred Dushin
Running a Solr query has no impact on writes -- Riak search queries are direct pass throughs to Solr query and don't touch any of the salient Riak systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of the reappearance is a coincidence.

Is it possible the object reappeared via MDC?  Do you have record of the reappeared object in your other cluster?

Also, just to confirm, when you say the object reappeared -- it re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying you just find the object in Solr (via a search)?

-Fred

On Feb 23, 2017, at 1:43 AM, al so <[hidden email]> wrote:

 Here is brief env:
 Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
 
 Problem:
  Deleted object suddenly resurrected after few days. Solr search query("*:*") was executed around the time of reappearance.

 Bucket property for this reappeared object
 {
  "props": {
    "name": "UsaHype",
    "allow_mult": false,
    "basic_quorum": false,
    "big_vclock": 50,
    "chash_keyfun": {
      "mod": "riak_core_util",
      "fun": "chash_std_keyfun"
    },
    "dvv_enabled": true,
    "dw": "quorum",
    "last_write_wins": false,
    "linkfun": {
      "mod": "riak_kv_wm_link_walker",
      "fun": "mapreduce_linkfun"
    },
    "n_val": 3,
    "notfound_ok": true,
    "old_vclock": 86400,
    "postcommit": [
      {
        "mod": "riak_repl_leader",
        "fun": "postcommit"
      },
      {
        "mod": "riak_repl2_rt",
        "fun": "postcommit"
      }
    ],
    "pr": 0,
    "precommit": [
      
    ],
    "pw": 0,
    "r": "quorum",
    "repl": true,
    "rw": "quorum",
    "small_vclock": 50,
    "w": "quorum",
    "young_vclock": 20
  }
}
 
 delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still get created when no delete_mode at all in the config? Can we query metadata(X-Riak-Deleted) for this reappeared object.
 How would one go about finding root cause. I do know the date when the object was deleted from our app logs. I also seem to know when it reappeared. Seems like a solr search ("*:*") in code was executed just before this object reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are out-of-sync (AAE?). 

 I understand tuning other params (allow_mult = true, ...) will have its own implications.

 In summary:
  How do I find root cause of this issue. 
  How to reliable delete an object.

-Volk

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak: reliable object deletion

al so
Present both in Solr and Riak. 

On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin <[hidden email]> wrote:
Running a Solr query has no impact on writes -- Riak search queries are direct pass throughs to Solr query and don't touch any of the salient Riak systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of the reappearance is a coincidence.

Is it possible the object reappeared via MDC?  Do you have record of the reappeared object in your other cluster?

Also, just to confirm, when you say the object reappeared -- it re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying you just find the object in Solr (via a search)?

-Fred

On Feb 23, 2017, at 1:43 AM, al so <[hidden email]> wrote:

 Here is brief env:
 Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
 
 Problem:
  Deleted object suddenly resurrected after few days. Solr search query("*:*") was executed around the time of reappearance.

 Bucket property for this reappeared object
 {
  "props": {
    "name": "UsaHype",
    "allow_mult": false,
    "basic_quorum": false,
    "big_vclock": 50,
    "chash_keyfun": {
      "mod": "riak_core_util",
      "fun": "chash_std_keyfun"
    },
    "dvv_enabled": true,
    "dw": "quorum",
    "last_write_wins": false,
    "linkfun": {
      "mod": "riak_kv_wm_link_walker",
      "fun": "mapreduce_linkfun"
    },
    "n_val": 3,
    "notfound_ok": true,
    "old_vclock": 86400,
    "postcommit": [
      {
        "mod": "riak_repl_leader",
        "fun": "postcommit"
      },
      {
        "mod": "riak_repl2_rt",
        "fun": "postcommit"
      }
    ],
    "pr": 0,
    "precommit": [
      
    ],
    "pw": 0,
    "r": "quorum",
    "repl": true,
    "rw": "quorum",
    "small_vclock": 50,
    "w": "quorum",
    "young_vclock": 20
  }
}
 
 delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still get created when no delete_mode at all in the config? Can we query metadata(X-Riak-Deleted) for this reappeared object.
 How would one go about finding root cause. I do know the date when the object was deleted from our app logs. I also seem to know when it reappeared. Seems like a solr search ("*:*") in code was executed just before this object reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are out-of-sync (AAE?). 

 I understand tuning other params (allow_mult = true, ...) will have its own implications.

 In summary:
  How do I find root cause of this issue. 
  How to reliable delete an object.

-Volk



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak: reliable object deletion

al so
Present both in Solr and Riak. It's a 1-way replication in MDC. MDC is not the cause unless there is a bug there as well.


On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin <[hidden email]> wrote:
Running a Solr query has no impact on writes -- Riak search queries are direct pass throughs to Solr query and don't touch any of the salient Riak systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of the reappearance is a coincidence.

Is it possible the object reappeared via MDC?  Do you have record of the reappeared object in your other cluster?

Also, just to confirm, when you say the object reappeared -- it re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying you just find the object in Solr (via a search)?

-Fred

On Feb 23, 2017, at 1:43 AM, al so <[hidden email]> wrote:

 Here is brief env:
 Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
 
 Problem:
  Deleted object suddenly resurrected after few days. Solr search query("*:*") was executed around the time of reappearance.

 Bucket property for this reappeared object
 {
  "props": {
    "name": "UsaHype",
    "allow_mult": false,
    "basic_quorum": false,
    "big_vclock": 50,
    "chash_keyfun": {
      "mod": "riak_core_util",
      "fun": "chash_std_keyfun"
    },
    "dvv_enabled": true,
    "dw": "quorum",
    "last_write_wins": false,
    "linkfun": {
      "mod": "riak_kv_wm_link_walker",
      "fun": "mapreduce_linkfun"
    },
    "n_val": 3,
    "notfound_ok": true,
    "old_vclock": 86400,
    "postcommit": [
      {
        "mod": "riak_repl_leader",
        "fun": "postcommit"
      },
      {
        "mod": "riak_repl2_rt",
        "fun": "postcommit"
      }
    ],
    "pr": 0,
    "precommit": [
      
    ],
    "pw": 0,
    "r": "quorum",
    "repl": true,
    "rw": "quorum",
    "small_vclock": 50,
    "w": "quorum",
    "young_vclock": 20
  }
}
 
 delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still get created when no delete_mode at all in the config? Can we query metadata(X-Riak-Deleted) for this reappeared object.
 How would one go about finding root cause. I do know the date when the object was deleted from our app logs. I also seem to know when it reappeared. Seems like a solr search ("*:*") in code was executed just before this object reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are out-of-sync (AAE?). 

 I understand tuning other params (allow_mult = true, ...) will have its own implications.

 In summary:
  How do I find root cause of this issue. 
  How to reliable delete an object.

-Volk




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak: reliable object deletion

al so
What is the best way to root cause it.

Any metadata/etc to query the lifecycle of this deleted object given that Key and Bucket are known. 

On Thu, Feb 23, 2017 at 9:53 PM, al so <[hidden email]> wrote:
Present both in Solr and Riak. It's a 1-way replication in MDC. MDC is not the cause unless there is a bug there as well.


On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin <[hidden email]> wrote:
Running a Solr query has no impact on writes -- Riak search queries are direct pass throughs to Solr query and don't touch any of the salient Riak systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of the reappearance is a coincidence.

Is it possible the object reappeared via MDC?  Do you have record of the reappeared object in your other cluster?

Also, just to confirm, when you say the object reappeared -- it re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying you just find the object in Solr (via a search)?

-Fred

On Feb 23, 2017, at 1:43 AM, al so <[hidden email]> wrote:

 Here is brief env:
 Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
 
 Problem:
  Deleted object suddenly resurrected after few days. Solr search query("*:*") was executed around the time of reappearance.

 Bucket property for this reappeared object
 {
  "props": {
    "name": "UsaHype",
    "allow_mult": false,
    "basic_quorum": false,
    "big_vclock": 50,
    "chash_keyfun": {
      "mod": "riak_core_util",
      "fun": "chash_std_keyfun"
    },
    "dvv_enabled": true,
    "dw": "quorum",
    "last_write_wins": false,
    "linkfun": {
      "mod": "riak_kv_wm_link_walker",
      "fun": "mapreduce_linkfun"
    },
    "n_val": 3,
    "notfound_ok": true,
    "old_vclock": 86400,
    "postcommit": [
      {
        "mod": "riak_repl_leader",
        "fun": "postcommit"
      },
      {
        "mod": "riak_repl2_rt",
        "fun": "postcommit"
      }
    ],
    "pr": 0,
    "precommit": [
      
    ],
    "pw": 0,
    "r": "quorum",
    "repl": true,
    "rw": "quorum",
    "small_vclock": 50,
    "w": "quorum",
    "young_vclock": 20
  }
}
 
 delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still get created when no delete_mode at all in the config? Can we query metadata(X-Riak-Deleted) for this reappeared object.
 How would one go about finding root cause. I do know the date when the object was deleted from our app logs. I also seem to know when it reappeared. Seems like a solr search ("*:*") in code was executed just before this object reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are out-of-sync (AAE?). 

 I understand tuning other params (allow_mult = true, ...) will have its own implications.

 In summary:
  How do I find root cause of this issue. 
  How to reliable delete an object.

-Volk





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak: reliable object deletion

al so
In reply to this post by al so
Implications when delete_mode set to 'keep'?


On Thu, Feb 23, 2017 at 9:53 PM, al so <[hidden email]> wrote:
Present both in Solr and Riak. It's a 1-way replication in MDC. MDC is not the cause unless there is a bug there as well.


On Thu, Feb 23, 2017 at 6:02 AM, Fred Dushin <[hidden email]> wrote:
Running a Solr query has no impact on writes -- Riak search queries are direct pass throughs to Solr query and don't touch any of the salient Riak systems (batching writes to Solr, YZ AAE, etc).  I believe the timing of the reappearance is a coincidence.

Is it possible the object reappeared via MDC?  Do you have record of the reappeared object in your other cluster?

Also, just to confirm, when you say the object reappeared -- it re-appeared in Riak (and subsequently in Solr), correct?  Or are you saying you just find the object in Solr (via a search)?

-Fred

On Feb 23, 2017, at 1:43 AM, al so <[hidden email]> wrote:

 Here is brief env:
 Riak v 2.0.8 + Solr/ 5 node cluster/ MDC
 
 Problem:
  Deleted object suddenly resurrected after few days. Solr search query("*:*") was executed around the time of reappearance.

 Bucket property for this reappeared object
 {
  "props": {
    "name": "UsaHype",
    "allow_mult": false,
    "basic_quorum": false,
    "big_vclock": 50,
    "chash_keyfun": {
      "mod": "riak_core_util",
      "fun": "chash_std_keyfun"
    },
    "dvv_enabled": true,
    "dw": "quorum",
    "last_write_wins": false,
    "linkfun": {
      "mod": "riak_kv_wm_link_walker",
      "fun": "mapreduce_linkfun"
    },
    "n_val": 3,
    "notfound_ok": true,
    "old_vclock": 86400,
    "postcommit": [
      {
        "mod": "riak_repl_leader",
        "fun": "postcommit"
      },
      {
        "mod": "riak_repl2_rt",
        "fun": "postcommit"
      }
    ],
    "pr": 0,
    "precommit": [
      
    ],
    "pw": 0,
    "r": "quorum",
    "repl": true,
    "rw": "quorum",
    "small_vclock": 50,
    "w": "quorum",
    "young_vclock": 20
  }
}
 
 delete_mode is not configured at all. Defaults to 3 sec? Do Tombstone still get created when no delete_mode at all in the config? Can we query metadata(X-Riak-Deleted) for this reappeared object.
 How would one go about finding root cause. I do know the date when the object was deleted from our app logs. I also seem to know when it reappeared. Seems like a solr search ("*:*") in code was executed just before this object reappeared? Look at AAE? Is there a way to find if Solr and Riak backends are out-of-sync (AAE?). 

 I understand tuning other params (allow_mult = true, ...) will have its own implications.

 In summary:
  How do I find root cause of this issue. 
  How to reliable delete an object.

-Volk





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...