Weird (?) error while using riak erlang pb client's mapreduce

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird (?) error while using riak erlang pb client's mapreduce

Alin Popa
Hi guys,

I've start using the mapreduce functionality from riak erlang's client (riak-erlang-client) on a riak node (not being part of a cluster) and I keep getting the follwing error in the crash.log file:

2012-01-06 16:43:17 =ERROR REPORT====
** State machine <0.1619.0> terminating 
** Last event in was {input,{{ok,{r_object,<<"small-bucket">>,<<"some-magnific-key-15">>,[{r_content,{dict,3,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"X-Riak-VTag">>,53,55,79,110,74,85,80,68,102,70,111,78,107,103,111,52,120,109,100,121,82,74]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1325,860978,730558}]],[],[]}}},<<"1970-01-01 05:27:00">>}],[{<<35,9,254,249,79,7,8,5>>,{1,63493080178}}],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean|true]],[]}}},undefined},undefined},[{936274486415109681974235595958868809467081785344,'[hidden email]'}]}}
** When State == wait_for_input
**      Data  == {state,936274486415109681974235595958868809467081785344,{fitting_details,{fitting,<0.1583.0>,#Ref<0.0.0.4840>,follow,1},0,riak_kv_mrc_map,{{qfun,#Fun<simple_mapreduce.0.110367059>},none},{fitting,<0.1581.0>,#Ref<0.0.0.4840>,sink,undefined},[{sink,{fitting,<0.1581.0>,#Ref<0.0.0.4840>,sink,undefined}},{log,sink},{trace,{set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[error],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}],64},<0.351.0>,{state,936274486415109681974235595958868809467081785344,{fitting_details,{fitting,<0.1583.0>,#Ref<0.0.0.4840>,follow,1},0,riak_kv_mrc_map,{{qfun,#Fun<simple_mapreduce.0.110367059>},none},{fitting,<0.1581.0>,#Ref<0.0.0.4840>,sink,undefined},[{sink,{fitting,<0.1581.0>,#Ref<0.0.0.4840>,sink,undefined}},{log,sink},{trace,{set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[error],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}],64},{qfun,#Fun<simple_mapreduce.0.110367059>},none}}
** Reason for termination = 
** processing_error
2012-01-06 16:43:17 =CRASH REPORT====
  crasher:
    initial call: riak_pipe_vnode_worker:init/1
    pid: <0.1619.0>
    registered_name: []
    exception exit: processing_error
      in function  gen_fsm:terminate/7
      in call from proc_lib:init_p_do_apply/3
    ancestors: [<0.352.0>,<0.351.0>,riak_core_vnode_sup,riak_core_sup,<0.75.0>]
    messages: []
    links: [<0.352.0>,<0.351.0>]
    dictionary: [{eunit,[{module,riak_pipe_vnode_worker},{partition,936274486415109681974235595958868809467081785344},{<0.351.0>,<0.351.0>},{details,{fitting_details,{fitting,<0.1583.0>,#Ref<0.0.0.4840>,follow,1},0,riak_kv_mrc_map,{{qfun,#Fun<simple_mapreduce.0.110367059>},none},{fitting,<0.1581.0>,#Ref<0.0.0.4840>,sink,undefined},[{sink,{fitting,<0.1581.0>,#Ref<0.0.0.4840>,sink,undefined}},{log,sink},{trace,{set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[error],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}],64}}]}]
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 621
  neighbours:
2012-01-06 16:43:17 =SUPERVISOR REPORT====
     Supervisor: {<0.352.0>,riak_pipe_vnode_worker_sup}
     Context:    child_terminated
     Reason:     processing_error
     Offender:   [{pid,<0.1619.0>},{name,undefined},{mfargs,{riak_pipe_vnode_worker,start_link,undefined}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]



and this is the code that I'm using for this:

-module(simple_mapreduce).

-compile(export_all).

main(Pid) ->
  {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
  Fun = fun(Object, _KeyData, none) -> [Object] end,
  riakc_pb_socket:mapred(Pid, {<<"small-bucket">>,[]}, [{map, {qfun, Count}, none, true}]).


It's a very simple usage, and according to the documentation, I haven't missed anything here (I know I can use the modfun instead of qfun for exactly this functionality, but it's just an example which throws).

Also, for the master version of riak, I'm getting error as return value of calling mapred function on riakc_pb_socket:

{error,<<"{\"phase\":0,\"error\":\"undef\",\"input\":\"{ok,{r_object,<<\\\"small-bucket\\\">>,<<\\\"some-magnific-key-44\\\">>,[{r_cont"...>>}

Is there a way that I can understand what these logs means ? It's all erlang information there, but the problem is that it's related to a lot of internals which I don't know.
Is it something very obvious that escapes my attention here ?

Thanks,
Alin


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Weird (?) error while using riak erlang pb client's mapreduce

bryan-basho
Administrator
On Fri, Jan 6, 2012 at 10:18 AM, Alin Popa <[hidden email]> wrote:

> and this is the code that I'm using for this:
>
> -module(simple_mapreduce).
>
> -compile(export_all).
>
> main(Pid) ->
>   {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
>   Fun = fun(Object, _KeyData, none) -> [Object] end,
>   riakc_pb_socket:mapred(Pid, {<<"small-bucket">>,[]}, [{map, {qfun, Count},
> none, true}]).
>
>
> It's a very simple usage, and according to the documentation, I haven't
> missed anything here (I know I can use the modfun instead of qfun for
> exactly this functionality, but it's just an example which throws).
>
> Also, for the master version of riak, I'm getting error as return value of
> calling mapred function on riakc_pb_socket:
>
> {error,<<"{\"phase\":0,\"error\":\"undef\",\"input\":\"{ok,{r_object,<<\\\"small-bucket\\\">>,<<\\\"some-magnific-key-44\\\">>,[{r_cont"...>>}
>
> Is there a way that I can understand what these logs means ?

The clue is the "error:'undef'" field in the error message JSON you
received.  Just like any undef error in an erlang program, it's
telling you the query tried to call a function that does not exist.

The reason the function does not exist is that the Riak cluster does
not have access to your 'simple_mapreduce' module.  Since the
anonymous function you are submitting was created in that module, Riak
needs to be able to load the same version of it in order to evaluate
the function.  If you had used modfun instead, you would have received
an error complaining that the module was unavailable.

To make your module available to Riak, point the 'add_paths' setting
in the riak_kv section of your app.config to the directory containing
your module's compiled beam file.

http://wiki.basho.com/Configuration-Files.html#add_paths

Also, be sure to reload the module on each riak cluster node (using
'riak-admin erl_reload' on the command line) after making changes and
recompiling your module.

Hope that helps,
Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Weird (?) error while using riak erlang pb client's mapreduce

Alin Popa
Hi Bryan,

Thanks, it makes sense, indeed, I've done what you've suggested, to load the code within riak (using add_paths) and everything worked fine. Still, I have a couple dilemas regarding this:

1. Why is not complaining if using escript instead of compiling it as a separate module (and no, I'm not loading the code within riak this time) ?

#!/usr/bin/env escript

main([]) ->
  code:add_patha("/Users/alin/riak-erlang-client/ebin"),
  code:add_patha("/Users/alin/riak-erlang-client/deps/protobuffs/ebin"),
  {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
  Fun = fun(Object, _KeyData, none) -> [Object] end,
  io:format("Result: ~n~p~n",[riakc_pb_socket:mapred(Pid, {<<"test-bucket">>,[]}, [{map, {qfun, Fun}, none, true}])]).

this might be an erlang thing ?

2. I remember that running exactly the same code in the past (~ 1 month ago), worked with no special intervention like adding code paths to riak. In fact, that's why I've raised this problem now, as it doesn't make any sense. Noticed this behavior on both release version (1.0.2) and master branch from github.

Thanks,
Alin 

On Fri, Jan 6, 2012 at 7:18 PM, Bryan Fink <[hidden email]> wrote:
On Fri, Jan 6, 2012 at 10:18 AM, Alin Popa <[hidden email]> wrote:
> and this is the code that I'm using for this:
>
> -module(simple_mapreduce).
>
> -compile(export_all).
>
> main(Pid) ->
>   {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
>   Fun = fun(Object, _KeyData, none) -> [Object] end,
>   riakc_pb_socket:mapred(Pid, {<<"small-bucket">>,[]}, [{map, {qfun, Count},
> none, true}]).
>
>
> It's a very simple usage, and according to the documentation, I haven't
> missed anything here (I know I can use the modfun instead of qfun for
> exactly this functionality, but it's just an example which throws).
>
> Also, for the master version of riak, I'm getting error as return value of
> calling mapred function on riakc_pb_socket:
>
> {error,<<"{\"phase\":0,\"error\":\"undef\",\"input\":\"{ok,{r_object,<<\\\"small-bucket\\\">>,<<\\\"some-magnific-key-44\\\">>,[{r_cont"...>>}
>
> Is there a way that I can understand what these logs means ?

The clue is the "error:'undef'" field in the error message JSON you
received.  Just like any undef error in an erlang program, it's
telling you the query tried to call a function that does not exist.

The reason the function does not exist is that the Riak cluster does
not have access to your 'simple_mapreduce' module.  Since the
anonymous function you are submitting was created in that module, Riak
needs to be able to load the same version of it in order to evaluate
the function.  If you had used modfun instead, you would have received
an error complaining that the module was unavailable.

To make your module available to Riak, point the 'add_paths' setting
in the riak_kv section of your app.config to the directory containing
your module's compiled beam file.

http://wiki.basho.com/Configuration-Files.html#add_paths

Also, be sure to reload the module on each riak cluster node (using
'riak-admin erl_reload' on the command line) after making changes and
recompiling your module.

Hope that helps,
Bryan


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Weird (?) error while using riak erlang pb client's mapreduce

Alin Popa
Hmmm, it seems that I've found why this is happening:


This problem was raised before and already solved, sorry for opening it again :)

Alin

On Fri, Jan 6, 2012 at 9:51 PM, Alin Popa <[hidden email]> wrote:
Hi Bryan,

Thanks, it makes sense, indeed, I've done what you've suggested, to load the code within riak (using add_paths) and everything worked fine. Still, I have a couple dilemas regarding this:

1. Why is not complaining if using escript instead of compiling it as a separate module (and no, I'm not loading the code within riak this time) ?

#!/usr/bin/env escript

main([]) ->
  code:add_patha("/Users/alin/riak-erlang-client/ebin"),
  code:add_patha("/Users/alin/riak-erlang-client/deps/protobuffs/ebin"),
  {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
  Fun = fun(Object, _KeyData, none) -> [Object] end,
  io:format("Result: ~n~p~n",[riakc_pb_socket:mapred(Pid, {<<"test-bucket">>,[]}, [{map, {qfun, Fun}, none, true}])]).

this might be an erlang thing ?

2. I remember that running exactly the same code in the past (~ 1 month ago), worked with no special intervention like adding code paths to riak. In fact, that's why I've raised this problem now, as it doesn't make any sense. Noticed this behavior on both release version (1.0.2) and master branch from github.

Thanks,
Alin 


On Fri, Jan 6, 2012 at 7:18 PM, Bryan Fink <[hidden email]> wrote:
On Fri, Jan 6, 2012 at 10:18 AM, Alin Popa <[hidden email]> wrote:
> and this is the code that I'm using for this:
>
> -module(simple_mapreduce).
>
> -compile(export_all).
>
> main(Pid) ->
>   {ok, Pid} = riakc_pb_socket:start_link("127.0.0.1", 8087),
>   Fun = fun(Object, _KeyData, none) -> [Object] end,
>   riakc_pb_socket:mapred(Pid, {<<"small-bucket">>,[]}, [{map, {qfun, Count},
> none, true}]).
>
>
> It's a very simple usage, and according to the documentation, I haven't
> missed anything here (I know I can use the modfun instead of qfun for
> exactly this functionality, but it's just an example which throws).
>
> Also, for the master version of riak, I'm getting error as return value of
> calling mapred function on riakc_pb_socket:
>
> {error,<<"{\"phase\":0,\"error\":\"undef\",\"input\":\"{ok,{r_object,<<\\\"small-bucket\\\">>,<<\\\"some-magnific-key-44\\\">>,[{r_cont"...>>}
>
> Is there a way that I can understand what these logs means ?

The clue is the "error:'undef'" field in the error message JSON you
received.  Just like any undef error in an erlang program, it's
telling you the query tried to call a function that does not exist.

The reason the function does not exist is that the Riak cluster does
not have access to your 'simple_mapreduce' module.  Since the
anonymous function you are submitting was created in that module, Riak
needs to be able to load the same version of it in order to evaluate
the function.  If you had used modfun instead, you would have received
an error complaining that the module was unavailable.

To make your module available to Riak, point the 'add_paths' setting
in the riak_kv section of your app.config to the directory containing
your module's compiled beam file.

http://wiki.basho.com/Configuration-Files.html#add_paths

Also, be sure to reload the module on each riak cluster node (using
'riak-admin erl_reload' on the command line) after making changes and
recompiling your module.

Hope that helps,
Bryan



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com