recursive/looping map-reduce

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

recursive/looping map-reduce

Michael Radford
I'm trying to figure out whether there's an easy way to accomplish
something like a "recursive/looping map-reduce query" in Riak.

A use case for this is when the objects in the database form a linked
list, tree, or graph of some kind, and you want to look up a set of
objects whose depth from the initial key set isn't known in advance.

For a simple example, say you have a log of events that are stored as
a linked list of blocks of some maximum size, and you want to retrieve
the most recent n events that have some property (which for whatever
reason isn't practical to index in advance).  It's easy to write a
function that looks up each block in order and stops when it has n
matches, but it would be nice to do this without O(n) roundtrips to
Riak and possibly having to discard a large fraction of the results on
the client side.

One hack solution would be to write a map-reduce query with some large
number of identical pairs of map and reduce phases, where once the
stopping condition is met, the subsequent phases will get a flag in
their input which makes them a cheap no-op.

Are there any other options short of running custom query-coordinating
code at the level of riak_pipe?

Thanks,
Mike

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: recursive/looping map-reduce

bryan-basho
Administrator
On Wed, Feb 22, 2012 at 8:23 PM, Michael Radford <[hidden email]> wrote:
> I'm trying to figure out whether there's an easy way to accomplish
> something like a "recursive/looping map-reduce query" in Riak.
>

>
> Are there any other options short of running custom query-coordinating
> code at the level of riak_pipe?

Hi, Mike.  There are no direct options for open-ended recursive input
in MR at this time.  Pipe does provide a very limited ability to do
so, but there is no external way to access Pipe directly at the
moment.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com