Moving mapred* functions out of riak_client?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Moving mapred* functions out of riak_client?

bryan-basho
Administrator
Hello, Community.

I'd like to know how many of you are directly using the mapred*
functions in the riak_client module (riak_client:mapred/2,3,4,
:mapred_stream/2,3,4, :mapred_bucket/2,3,4, etc.). Of those that are,
how many of you would mind if they moved out of riak_client and into a
separate module?

Just to be clear: if you're accessing Riak's MapReduce system over
HTTP or ProtocolBuffers, this change would not affect you. My search
is only for people using the native distributed Erlang client
(https://github.com/basho/riak_kv/blob/master/src/riak_client.erl).

I'm doing some other work on Riak's MapReduce system, and as part of
the cleanup, I'd like to deprecate the riak_client:mapred* functions,
and replace them with functions in another module.  Basically, any
code you had like this:

   {ok, C} = riak:local_client(),
   {ok, Results} = C:mapred(Inputs, Query).

would need to be converted to this:

   {ok, Results} = riak_kv_mapred_iface:mapred(Inputs, Query).

Additionally, remote connections that looked like this:

   {ok, C} = riak:client_connect(RiakNode),
   {ok, Results} = C:mapred(Inputs, Query).

would need to be converted to something like this:

   {ok, Results} = rpc:call(RiakNode, riak_kv_mapred_iface, mapred,
[Inputs, Query]).

Would this change be distasteful or unbearable for any of you?

Thanks,

Bryan Fink
Senior Software Engineer
Basho Technologies

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Moving mapred* functions out of riak_client?

Kresten Krab Thorup
We are using those APIs in our new stuff for riak sync/mobile. My problem with your change would be that we have a mock riak_client which is super convenient for unit testing, and it would be good to still be able to have that somehow.  Our mock is just an eta-backed parameterized module which has put/get/mapred_bucket/etc. ....

Kresten

On 10/06/2011, at 12.33, "Bryan Fink" <[hidden email]> wrote:

> Hello, Community.
>
> I'd like to know how many of you are directly using the mapred*
> functions in the riak_client module (riak_client:mapred/2,3,4,
> :mapred_stream/2,3,4, :mapred_bucket/2,3,4, etc.). Of those that are,
> how many of you would mind if they moved out of riak_client and into a
> separate module?
>
> Just to be clear: if you're accessing Riak's MapReduce system over
> HTTP or ProtocolBuffers, this change would not affect you. My search
> is only for people using the native distributed Erlang client
> (https://github.com/basho/riak_kv/blob/master/src/riak_client.erl).
>
> I'm doing some other work on Riak's MapReduce system, and as part of
> the cleanup, I'd like to deprecate the riak_client:mapred* functions,
> and replace them with functions in another module.  Basically, any
> code you had like this:
>
>   {ok, C} = riak:local_client(),
>   {ok, Results} = C:mapred(Inputs, Query).
>
> would need to be converted to this:
>
>   {ok, Results} = riak_kv_mapred_iface:mapred(Inputs, Query).
>
> Additionally, remote connections that looked like this:
>
>   {ok, C} = riak:client_connect(RiakNode),
>   {ok, Results} = C:mapred(Inputs, Query).
>
> would need to be converted to something like this:
>
>   {ok, Results} = rpc:call(RiakNode, riak_kv_mapred_iface, mapred,
> [Inputs, Query]).
>
> Would this change be distasteful or unbearable for any of you?
>
> Thanks,
>
> Bryan Fink
> Senior Software Engineer
> Basho Technologies
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Moving mapred* functions out of riak_client?

bryan-basho
Administrator
On Fri, Jun 10, 2011 at 11:18 AM, Kresten Krab Thorup <[hidden email]> wrote:
> We are using those APIs in our new stuff for riak sync/mobile. My problem with your change would be that we have a mock riak_client which is super convenient for unit testing, and it would be good to still be able to have that somehow.  Our mock is just an eta-backed parameterized module which has put/get/mapred_bucket/etc. ....

Hi, Kresten.  I can see how unit tests like you describe could be more
challenging with the breakup I proposed.

In case the link isn't obvious now, this is the driver behind my
desire to move things around (Riak Pipe):
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004550.html

My main thought was that keeping all of the code separate, right up to
the external interface layer, would reduce confusion during any
transition from the legacy system to the pipe-based system.

Would you mind taking a look at the changes I was thinking about to
support this inside Riak?  I have a branch + [unreviewed] pull request
going here: https://github.com/beerriot/riak_kv/pull/8

Or if I could take a look at your mock system, maybe I could help with
some suggestions for moving in the direction I proposed?  We are
generally trying to move toward more well-defined APIs (such as
PBC/HTTP) and to discourage people from using distributed Erlang to
interact with Riak.

Thanks,
Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Moving mapred* functions out of riak_client?

Les Mikesell
On 6/13/2011 2:40 PM, Bryan Fink wrote:
>
> Or if I could take a look at your mock system, maybe I could help with
> some suggestions for moving in the direction I proposed?  We are
> generally trying to move toward more well-defined APIs (such as
> PBC/HTTP) and to discourage people from using distributed Erlang to
> interact with Riak.

Are these going to expose any of the server distribution to clients
using PBC/HTTP?  It just seems odd that an inherently
distributed/masterless system requires an intermediate load
balancer/fail over mechanism instead of having clients that know/learn
how to round-robin and retry on their own.

--
   Les Mikesell
    [hidden email]


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Moving mapred* functions out of riak_client?

bryan-basho
Administrator
On Mon, Jun 13, 2011 at 4:01 PM, Les Mikesell <[hidden email]> wrote:

> On 6/13/2011 2:40 PM, Bryan Fink wrote:
>>
>> Or if I could take a look at your mock system, maybe I could help with
>> some suggestions for moving in the direction I proposed?  We are
>> generally trying to move toward more well-defined APIs (such as
>> PBC/HTTP) and to discourage people from using distributed Erlang to
>> interact with Riak.
>
> Are these going to expose any of the server distribution to clients using
> PBC/HTTP?  It just seems odd that an inherently distributed/masterless
> system requires an intermediate load balancer/fail over mechanism instead of
> having clients that know/learn how to round-robin and retry on their own.

We certainly have the desire to expose those features.  Just a matter
of priorities.

I'll note that the erlang-distribution-based riak_client module being
discussed in this thread does not expose any of these features (load
balancing, failover) either, so nothing is "lost" by moving to the
others.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com