Using mapreduce in a webmachine app?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Using mapreduce in a webmachine app?

Evan Haas
I'm running into a bit of trouble using the riak_client mapreduce functionality from within a webmachine resource.  The strange thing is that it works fine from a standalone client running in an erlang shell.  Here's the code:

{ok, Client} = riak:client_connect('[hidden email]'),
Keys = [{<<"bucket">>, <<"key">>],
GetObjects = fun(G, _Data, _Arg) -> [riak_object:get_value(G)] end,
Response = Client:mapred(Keys, [{map, {qfun, GetObjects}, none, true}], 10000).

Like I said, if I run that from an erlang shell, it does the right thing.  However if I run it from within a webmachine resource, the return value from Client:mapred is "all nodes failed".  FWIW regular get/put/delete operations work fine from within my resource, it's only mapreduce that seems to be having a problem.  Furthermore, if I use riak_mapreduce:map_object_value(true) instead of constructing my own spec/function, it works properly both within a shell and from webmachine.  Any thoughts?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Using mapreduce in a webmachine app?

Ben Browning
I ran into this problem a week ago and after some help in IRC we
determined that qfun map/reduce phases aren't working properly if run
from a remote Erlang node. As a workaround I had to use modfuns.

When your qfun example works, is the erlang shell you're running it in
the one given by running riak attach or riak console? I found qfuns
worked properly in that case.

The only pain about modfuns is you need to load your code on all the
nodes. When testing things I find it easiest to do this like so:

% connect to riak and pause so nodes() gets populated
{ok, Client} = riak:client_connect('riak@127.0.0.1').
timer:sleep(100).
% replace my_mapreduce_module with your module's name
{Mod, Bin, File} = code:get_object_code(my_mapreduce_module).
{Replies, _} = rpc:multicall(code, load_binary, [Mod, File, Bin]).

This loads your custom module code in all running Riak nodes without
having to copy files around or restart the nodes.

Hope that helps,

Ben

On Sun, Mar 28, 2010 at 4:42 PM, Evan Haas <[hidden email]> wrote:

> I'm running into a bit of trouble using the riak_client mapreduce
> functionality from within a webmachine resource.  The strange thing is that
> it works fine from a standalone client running in an erlang shell.  Here's
> the code:
>
> {ok, Client} = riak:client_connect('riak@127.0.0.1'),
> Keys = [{<<"bucket">>, <<"key">>],
> GetObjects = fun(G, _Data, _Arg) -> [riak_object:get_value(G)] end,
> Response = Client:mapred(Keys, [{map, {qfun, GetObjects}, none, true}],
> 10000).
>
> Like I said, if I run that from an erlang shell, it does the right thing.
> However if I run it from within a webmachine resource, the return value from
> Client:mapred is "all nodes failed".  FWIW regular get/put/delete operations
> work fine from within my resource, it's only mapreduce that seems to be
> having a problem.  Furthermore, if I use
> riak_mapreduce:map_object_value(true) instead of constructing my own
> spec/function, it works properly both within a shell and from webmachine.
> Any thoughts?
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Using mapreduce in a webmachine app?

Evan Haas
Thanks, that solved the problem.

On Sun, Mar 28, 2010 at 2:58 PM, Ben Browning <[hidden email]> wrote:
I ran into this problem a week ago and after some help in IRC we
determined that qfun map/reduce phases aren't working properly if run
from a remote Erlang node. As a workaround I had to use modfuns.

When your qfun example works, is the erlang shell you're running it in
the one given by running riak attach or riak console? I found qfuns
worked properly in that case.

The only pain about modfuns is you need to load your code on all the
nodes. When testing things I find it easiest to do this like so:

% connect to riak and pause so nodes() gets populated
{ok, Client} = riak:client_connect('[hidden email]').
timer:sleep(100).
% replace my_mapreduce_module with your module's name
{Mod, Bin, File} = code:get_object_code(my_mapreduce_module).
{Replies, _} = rpc:multicall(code, load_binary, [Mod, File, Bin]).

This loads your custom module code in all running Riak nodes without
having to copy files around or restart the nodes.

Hope that helps,

Ben

On Sun, Mar 28, 2010 at 4:42 PM, Evan Haas <[hidden email]> wrote:
> I'm running into a bit of trouble using the riak_client mapreduce
> functionality from within a webmachine resource.  The strange thing is that
> it works fine from a standalone client running in an erlang shell.  Here's
> the code:
>
> {ok, Client} = riak:client_connect('[hidden email]'),
> Keys = [{<<"bucket">>, <<"key">>],
> GetObjects = fun(G, _Data, _Arg) -> [riak_object:get_value(G)] end,
> Response = Client:mapred(Keys, [{map, {qfun, GetObjects}, none, true}],
> 10000).
>
> Like I said, if I run that from an erlang shell, it does the right thing.
> However if I run it from within a webmachine resource, the return value from
> Client:mapred is "all nodes failed".  FWIW regular get/put/delete operations
> work fine from within my resource, it's only mapreduce that seems to be
> having a problem.  Furthermore, if I use
> riak_mapreduce:map_object_value(true) instead of constructing my own
> spec/function, it works properly both within a shell and from webmachine.
> Any thoughts?
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com