PUT behavior when allow_mult=true differs based on client ID?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

PUT behavior when allow_mult=true differs based on client ID?

Robert Horvick
I'm noticing what appears to be an odd behavior when uploading conflicts.

I have a bucket with allow_mult:true.  When I add the first key (non-conflicting) everything works as expected (HTTP 204 - I didn't request content back) - the key is added.  When I add the second key value (the first conflict) I again get HTTP 204 and now there is a conflict (querying the key gives HTTP/300 and displays two siblings.  However adding subsequent key values continue to return HTTP 204 but they don't show up in the sibling list (the list stays at exactly two items).

It is very confusing to me that four PUTs on the same key would result in only two siblings.  Oddly enough, if I don't specify an client ID header then all four siblings are created and it works as I expected.

Here is an example curl session (sorry for the guids - this is coming out of my test cases so guids are what get used):

curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "axvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "bxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "cxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "dxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5



curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "axvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "bxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "cxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "dxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6



Notice that the GET on key "multi5" returns two siblings and key "multi6" returns four.  The only difference between the curl statements is the absence of the X-Riak-ClientId header.

Is this expected?  If so - why?

Thanks,

Robert.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Fwd: PUT behavior when allow_mult=true differs based on client ID?

dreverri
As far as I know this was implemented to reduce the length of the vector clocks. As updates are made the vector clock length will be increased to accomodate the version change. By specifying the client id riak can assume the client making the request knows about the conflicting versions and can ignore the out of date ones. Out of date in this case means that the client is submitting values based on a version it has already updated.

client1 -> update version1 -> riak creates version2
client1 -> update version2 -> riak creates version3
client1 -> update version1 -> riak notes that client1 has already updated version1 and can drop this version as out of date

By not specifying the client id riak cannot make an assumption about what the submitting client knows and will accept the sibling.





On Tue, Mar 23, 2010 at 3:52 PM, Robert Horvick <[hidden email]> wrote:
I'm noticing what appears to be an odd behavior when uploading conflicts.

I have a bucket with allow_mult:true.  When I add the first key (non-conflicting) everything works as expected (HTTP 204 - I didn't request content back) - the key is added.  When I add the second key value (the first conflict) I again get HTTP 204 and now there is a conflict (querying the key gives HTTP/300 and displays two siblings.  However adding subsequent key values continue to return HTTP 204 but they don't show up in the sibling list (the list stays at exactly two items).

It is very confusing to me that four PUTs on the same key would result in only two siblings.  Oddly enough, if I don't specify an client ID header then all four siblings are created and it works as I expected.

Here is an example curl session (sorry for the guids - this is coming out of my test cases so guids are what get used):

curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "axvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "bxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "cxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "dxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5



curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "axvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "bxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "cxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6
curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H "Content-Type: text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data "dxvqkrzcblgrsczqacv" http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi6



Notice that the GET on key "multi5" returns two siblings and key "multi6" returns four.  The only difference between the curl statements is the absence of the X-Riak-ClientId header.

Is this expected?  If so - why?

Thanks,

Robert.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: PUT behavior when allow_mult=true differs based on client ID?

bryan-basho
Administrator
In reply to this post by Robert Horvick
Hi, Robert.  The behavior you are seeing is expected.  The short
answer is that each concurrent actor in the system should be using a
unique client ID.  Given that your four example requests were intended
to simulate concurrency (because otherwise they would have included
updated X-Riak-Vclock headers), each should have used a different
value for X-Riak-ClientId.  For the longer answer, read on.

Omitting a vector clock from a PUT request is the same as using an
empty vector clock.  So, each of your requests is effectively saying,
"This is the first version of this object."

Client IDs come into play while Riak is processing your request.  When
Riak receives a PUT, it increments the vector clock in the request
using the client ID from the request.  If there is no client ID
included with the request, Riak chooses a random client ID for the
increment.

So, if we take a look at your requests, one by one:

On Tue, Mar 23, 2010 at 6:52 PM, Robert Horvick
<[hidden email]> wrote:
> curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H
> "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type:
> text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data
> "axvqkrzcblgrsczqacv"

Your first update, says:
vclock: <<empty>>
clientid: b88d

Riak updates the vclock before storing your value:
new vclock: b88d:1

> http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
> curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H
> "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type:
> text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data
> "bxvqkrzcblgrsczqacv"

Your second update also says:
vclock: <<empty>>
clientid: b88d

Riak updates the vclock before storing your value:
new vclock: b88d:1

Now when Riak goes to write this value, it finds that the vclock is
exactly equal to the vclock that is already stored.  Here a bit of
"smarts" kicks in, and Riak checks to see if the rest of the object is
equal as well.  Since it's not, Riak writes both versions of the
object as siblings.

The key to the subsequent confusing behavior is that before Riak
writes the sibling-containing object, it increments the vector clock
one more time, using a new actor.  So, what is actually on-disk now
is:
really new vclock: b88d:1, XXX:1

"XXX" being that new actor.  This is done to signify that there is
another actor touching the data, and that this update should be
considered independent of any other update happening in the system.

> http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
> curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H
> "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type:
> text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data
> "cxvqkrzcblgrsczqacv"

So, now your third request, again saying:
vclock: <<empty>>
clientid: b88d

Riak again updates the vclock before storing your value:
new vclock: b88d:1

This time, when Riak compares this vclock to the vclock it has stored,
it finds that the stored vclock supersedes this request vclock
(because the b88d count is equal, and the XXX count is greater).
Since the stored vclock is already "later" (logically) than the
request vclock, the value of the request is thrown out.

The same thing happens during your fourth request: the stored vclock
is found to be later than the request vclock, so the value of the
request is discarded.

The requests you tried that do not include an X-Riak-ClientId header
do not have this issue because they chose a new, random ID for each
request.  This means that none of the vector clocks involved ever
supersede each other, so siblings are always created.

Indeed, the randomly-generated client IDs illustrate the proper way to
manage this behavior: use a unique ID for each actor in the system.
The purpose of client IDs is to give each actor a way to update the
vector clock of an object, without any chance of stomping on
modifications by other actors.

Given that your four requests were intended to simulate concurrency
(because serial updates should have included X-Riak-Vclock headers
specifying updated vector clocks), each of them should have been using
a separate client ID.

Does all that make sense?

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: PUT behavior when allow_mult=true differs based on client ID?

Robert Horvick
Bryan,

Thank you very much for the detailed explanation - it was very helpful both with this specific issue and my overall understanding of vclocks and conflict handling.

Robert.

On Wed, Mar 24, 2010 at 9:43 AM, Bryan Fink <[hidden email]> wrote:
Hi, Robert.  The behavior you are seeing is expected.  The short
answer is that each concurrent actor in the system should be using a
unique client ID.  Given that your four example requests were intended
to simulate concurrency (because otherwise they would have included
updated X-Riak-Vclock headers), each should have used a different
value for X-Riak-ClientId.  For the longer answer, read on.

Omitting a vector clock from a PUT request is the same as using an
empty vector clock.  So, each of your requests is effectively saying,
"This is the first version of this object."

Client IDs come into play while Riak is processing your request.  When
Riak receives a PUT, it increments the vector clock in the request
using the client ID from the request.  If there is no client ID
included with the request, Riak chooses a random client ID for the
increment.

So, if we take a look at your requests, one by one:

On Tue, Mar 23, 2010 at 6:52 PM, Robert Horvick
<[hidden email]> wrote:
> curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H
> "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type:
> text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data
> "axvqkrzcblgrsczqacv"

Your first update, says:
vclock: <<empty>>
clientid: b88d

Riak updates the vclock before storing your value:
new vclock: b88d:1

> http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
> curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H
> "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type:
> text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data
> "bxvqkrzcblgrsczqacv"

Your second update also says:
vclock: <<empty>>
clientid: b88d

Riak updates the vclock before storing your value:
new vclock: b88d:1

Now when Riak goes to write this value, it finds that the vclock is
exactly equal to the vclock that is already stored.  Here a bit of
"smarts" kicks in, and Riak checks to see if the rest of the object is
equal as well.  Since it's not, Riak writes both versions of the
object as siblings.

The key to the subsequent confusing behavior is that before Riak
writes the sibling-containing object, it increments the vector clock
one more time, using a new actor.  So, what is actually on-disk now
is:
really new vclock: b88d:1, XXX:1

"XXX" being that new actor.  This is done to signify that there is
another actor touching the data, and that this update should be
considered independent of any other update happening in the system.

> http://192.168.137.50:8098/riak/32fa9a49-411e-4cc9-a5d5-0adc28865f62/multi5
> curl -X PUT -H "Accept: */*" -H "User-Agent: Riak Client for .NET" -H
> "X-Riak-ClientId: b88de040-8067-4a83-a0fb-829095aec01f" -H "Content-Type:
> text/plain" -H "Content-Length: 18" -H "Expect: 100-continue" --data
> "cxvqkrzcblgrsczqacv"

So, now your third request, again saying:
vclock: <<empty>>
clientid: b88d

Riak again updates the vclock before storing your value:
new vclock: b88d:1

This time, when Riak compares this vclock to the vclock it has stored,
it finds that the stored vclock supersedes this request vclock
(because the b88d count is equal, and the XXX count is greater).
Since the stored vclock is already "later" (logically) than the
request vclock, the value of the request is thrown out.

The same thing happens during your fourth request: the stored vclock
is found to be later than the request vclock, so the value of the
request is discarded.

The requests you tried that do not include an X-Riak-ClientId header
do not have this issue because they chose a new, random ID for each
request.  This means that none of the vector clocks involved ever
supersede each other, so siblings are always created.

Indeed, the randomly-generated client IDs illustrate the proper way to
manage this behavior: use a unique ID for each actor in the system.
The purpose of client IDs is to give each actor a way to update the
vector clock of an object, without any chance of stomping on
modifications by other actors.

Given that your four requests were intended to simulate concurrency
(because serial updates should have included X-Riak-Vclock headers
specifying updated vector clocks), each of them should have been using
a separate client ID.

Does all that make sense?

-Bryan


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com