Riak mapreduce function map_object_value return empty result on erlang pb client

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak mapreduce function map_object_value return empty result on erlang pb client

Mike_Kuznetsov

I use riak erlang client for my project, and update riak and client library today from master(client to 1.3.1 and server to 1.2.0). Everything works except fetching multiple entities with map-reduce return empty. Instead of usual result {ok, [...,{<stage_number>,<stage result>},...]} I get {ok,[]}. Rollback with client and everything works ok. But if I use old client with new riak I got a lot of errors in the server logs.

This is a code example, where BK is list of {Bucket, Key} pairs and every pair is related to an existing entity:

{ok, [{0,Result}]}=riakc_pb_socket:mapred(Pid, BK,[{map, {modfun, riak_kv_mapreduce, map_object_value},undefined, true}])

How can I change that line of code to make it work?



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak mapreduce function map_object_value return empty result on erlang pb client

bryan-basho
Administrator
On Thu, Oct 11, 2012 at 1:32 AM, Mikhail Kuznetsov
<[hidden email]> wrote:
> I use riak erlang client for my project, and update riak and client library
> today from master(client to 1.3.1 and server to 1.2.0). Everything works
> except fetching multiple entities with map-reduce return empty. Instead of
> usual result {ok, [...,{<stage_number>,<stage result>},...]} I get {ok,[]}.
> Rollback with client and everything works ok. But if I use old client with
> new riak I got a lot of errors in the server logs.

Hi, Mikhail. I'm not surprised by the old-client-with-new-riak errors.
The definitions of some protocol buffers changed in the 1.2.0 Riak
release, iirc.

As for the missing results, though, I've been unable to reproduce the
issue as you reported. Using Riak 1.2.0 and riak-erlang-client 1.3.1:


1> {ok, S} = riakc_pb_socket:start("127.0.0.1", 8087).
{ok,<0.33.0>}

2> BK = [{<<"mk">>,<<"1">>},{<<"mk">>,<<"2">>},{<<"mk">>,<<"3">>}].
[{<<"mk">>,<<"1">>},{<<"mk">>,<<"2">>},{<<"mk">>,<<"3">>}]

3> [ riakc_pb_socket:put(S, riakc_obj:new(B, K, K)) || {B, K} <- BK ].
[ok,ok,ok]

4> {ok, [{0,Result}]} = riakc_pb_socket:mapred(S, BK, [{map, {modfun,
riak_kv_mapreduce, map_object_value},undefined, true}]).
{ok,[{0,[<<"1">>,<<"2">>,<<"3">>]}]}

Can you see where my example differs from yours?

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak mapreduce function map_object_value return empty result on erlang pb client

Mike_Kuznetsov
Hi 
Thank you Brian. Today we finally find problems 
We use index fetch before use mapreduce, like in this example:

{ok,List}=riakc_pb_socket:get_index(Pid, Bucket, Index, Key)
BK=[ {B,K}||[B,K]<-List],
{ok,[{0,Result}]}=riakc_pb_socket:mapred(Pid, BK, [{map, {modfun, riak_kv_mapreduce, map_object_value} ,undefined, true}]),
[binary_to_term(E)|| E<-Result]

In line 1 function return list of lists (pair Bucket and Key). And after some time that function start returning simply list of keys. Without announcement, release notes or ticket or something else. Just one day one man think why not to rewrite everything related to search. Right before it someone "fixed" protobuff and we got tons of errors related to binary indexes. What the fuck, we spend one day to check driver source for possible bugs? I think main commiter at least should post some notes and not to change function interfaces without some serious need. 

Erlang VM in some case use it pattern matching without runtime error in line 3 and we got our silent bug. All we need was to change line 3 to
BK=[ {Bucket,K} || K<-List],

It works fine now. But this is not ok. Because interface is broke for map reduce function. We pass it empty list instead of list of {Bucket, Key} and we get {ok,[]} instead of {ok, [{0,[]}]}.
We fixed it, but I am very angry to people who don't give a fuck about interface stability and release notes. Many people use it in production, they pay money for support and a lot of money. They don't want to play in hide and seek "new features" after each minor commit. If they feel pissed they just pull out  they money and buy something else.
-- 
Sincerely yours,
Mikhail Kuznetsov

When best practices meet everyday life and lead to perfection...

Oct 15, 2012, в 19:55 , Bryan Fink написал(а):

On Thu, Oct 11, 2012 at 1:32 AM, Mikhail Kuznetsov
<[hidden email]> wrote:
I use riak erlang client for my project, and update riak and client library
today from master(client to 1.3.1 and server to 1.2.0). Everything works
except fetching multiple entities with map-reduce return empty. Instead of
usual result {ok, [...,{<stage_number>,<stage result>},...]} I get {ok,[]}.
Rollback with client and everything works ok. But if I use old client with
new riak I got a lot of errors in the server logs.

Hi, Mikhail. I'm not surprised by the old-client-with-new-riak errors.
The definitions of some protocol buffers changed in the 1.2.0 Riak
release, iirc.

As for the missing results, though, I've been unable to reproduce the
issue as you reported. Using Riak 1.2.0 and riak-erlang-client 1.3.1:


1> {ok, S} = riakc_pb_socket:start("127.0.0.1", 8087).
{ok,<0.33.0>}

2> BK = [{<<"mk">>,<<"1">>},{<<"mk">>,<<"2">>},{<<"mk">>,<<"3">>}].
[{<<"mk">>,<<"1">>},{<<"mk">>,<<"2">>},{<<"mk">>,<<"3">>}]

3> [ riakc_pb_socket:put(S, riakc_obj:new(B, K, K)) || {B, K} <- BK ].
[ok,ok,ok]

4> {ok, [{0,Result}]} = riakc_pb_socket:mapred(S, BK, [{map, {modfun,
riak_kv_mapreduce, map_object_value},undefined, true}]).
{ok,[{0,[<<"1">>,<<"2">>,<<"3">>]}]}

Can you see where my example differs from yours?

-Bryan


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak mapreduce function map_object_value return empty result on erlang pb client

bryan-basho
Administrator
On Mon, Oct 15, 2012 at 12:34 PM, Mikhail Kuznetsov
<[hidden email]> wrote:
> We fixed it, but I am very angry…

I agree. Breaking APIs without announcement is not okay. In this case,
I don't think it was intentional. Instead, I think we just didn't know
we broke this until your message arrived. In the latest version, we
changed get_index to use a new protocol buffer message instead of
piggy-backing on the MapReduce message. At the time this change was
made, no tests covered riakc_pb_socket:get_index:

https://github.com/basho/riak-erlang-client/blob/a33a677e393b90c88af2c05f4178984e3b634302/src/riakc_pb_socket.erl#L1398-2307

Without tests in place, we missed the fact that we changed the API. I
hope it improves your outlook to know that tests for get_index are now
included in the new riak_test project:

https://github.com/basho/riak_test/blob/master/tests/secondary_index_tests.erl

So, this kind of unknown, unexpected breakage should not happen again.

Thanks for spending time tracking this down, and sending in the report.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com