Using map 'arg' in mapreduce calls in python and Erlang clients

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Using map 'arg' in mapreduce calls in python and Erlang clients

Julian
Hi,

I have an Erlang module added to the Riak cluster via the add_paths config. It has some custom functions for map/reduce phases. In one of them I'm using the map argument to pass some configuration information.

At first, I was just developing with the Erlang client, so I was passing a proplist to riakc_pb_socket:mapred. That worked great. However, I also have to support Python here. I thought for a second and figured that I can't expect to be able to send an Erlang-specific data format from Python, so I changed my map function to expect a binary JSON blob. It seems like a binary would be the simplest thing to get working for Python. However, sending a string doesn't get the arg as a binary to the map function. I have an is_binary guard to the function,

map_reindex(RiakObject, _KeyData, IndexJson) when is_binary(IndexJson) ->

and the error that the python client gets starts with Exception: {"phase":0,"error":"function_clause"

so I don't think it's being received as a binary.

And how do I print out some logging information from my map function? I tried using lager like I found in the riak code,

    lager:error("Arg is ~p", [IndexJson]),

but I can't find the message in any of the Riak logs.

Thanks,
Julian

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Using map 'arg' in mapreduce calls in python and Erlang clients

bryan-basho
Administrator
On Fri, Nov 9, 2012 at 2:48 PM, Julian <[hidden email]> wrote:
> map_reindex(RiakObject, _KeyData, IndexJson) when is_binary(IndexJson) ->
>
> and the error that the python client gets starts with Exception:
> {"phase":0,"error":"function_clause"
>
> so I don't think it's being received as a binary.

Hi, Julian. You're right, it's not being received as a binary. The
MapReduce endpoint expects the 'arg' to be JSON, and decodes it. If
you're constructing your 'arg' like so:

    client.map(['module','function'], {'arg':'foobar'})

then I would expect the binary <<"foobar">> to come through (indeed,
it does in my testing). But, if you're constructing like so:

    client.map(['module','function'], {'arg':{'foobar':1}})

then instead, you'll get a mochijson2-decoded structure. That looks
something like this in Erlang:

    {struct, [{<<"foobar">>, 1}]}

More detail about that decoding can be found here:
https://github.com/mochi/mochiweb/blob/master/src/mochijson2.erl

> And how do I print out some logging information from my map function? I
> tried using lager like I found in the riak code,
>
>     lager:error("Arg is ~p", [IndexJson]),
>
> but I can't find the message in any of the Riak logs.

Lager requires a parse transform at compile time. I suggest using the
error_logger module. The lager application will pick up any calls to
make to error_logger:info_message, :warning_msg, and :error_msg, and
dump them in your console.log or error.log, as appropriate.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com