MapReduce paging result

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

MapReduce paging result

jason
Hi guys,
    How to do a paging in MapReduce queries?
    I try to write some  method in the module(trend_riak.erl, Add  -pa /home/jason/work/server/trunk/trend_riak    to vm.args).
    I thought it will well, but unfortunately it does not work. Becase the "Reduce" ,the result is wrong.
    Is there the other methods? Thanks.

   Cheers Jason
    
121 get_reduce_bucket_values(Records, {Start, Max}) ->
122     Skip = get_skip_num(Start, Max),
123     io:format("get_reduce_bucket_values Records=~p Skip= ~p~n", [length(Records), Skip]),                                                        
124     if length(Records) >= Skip  ->
125             %%NewList = lists:keysort(#pubsub_item.published, Records),
126             F = fun(#pubsub_item{published=Published1, level=Level1},
127                     #pubsub_item{published=Published2, level=Level2}) ->
128                     if Level1 > Level2 -> true;
129                        Level1 == Level2 ->
130                             if  Published1 > Published2 -> true;
131                                 true -> false
132                             end;
133                        true -> false
134                     end
135                 end,
136             NewList = lists:sort(F, Records),
137             R = lists:sublist(NewList, Skip, Max),
138             io:format("get_reduce_bucket_value R=~p Skip= ~p~n", [length(R), skip]),
139             R;
140        true ->
141             []
142             %Records 
143     end.


--
只为成功找方法,不为失败找理由

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: MapReduce paging result

jason
Who can answer me?  thanks.

2012/8/10 郎咸武 <[hidden email]>
Hi guys,
    How to do a paging in MapReduce queries?
    I try to write some  method in the module(trend_riak.erl, Add  -pa /home/jason/work/server/trunk/trend_riak    to vm.args).
    I thought it will well, but unfortunately it does not work. Becase the "Reduce" ,the result is wrong.
    Is there the other methods? Thanks.

   Cheers Jason
    
121 get_reduce_bucket_values(Records, {Start, Max}) ->
122     Skip = get_skip_num(Start, Max),
123     io:format("get_reduce_bucket_values Records=~p Skip= ~p~n", [length(Records), Skip]),                                                        
124     if length(Records) >= Skip  ->
125             %%NewList = lists:keysort(#pubsub_item.published, Records),
126             F = fun(#pubsub_item{published=Published1, level=Level1},
127                     #pubsub_item{published=Published2, level=Level2}) ->
128                     if Level1 > Level2 -> true;
129                        Level1 == Level2 ->
130                             if  Published1 > Published2 -> true;
131                                 true -> false
132                             end;
133                        true -> false
134                     end
135                 end,
136             NewList = lists:sort(F, Records),
137             R = lists:sublist(NewList, Skip, Max),
138             io:format("get_reduce_bucket_value R=~p Skip= ~p~n", [length(R), skip]),
139             R;
140        true ->
141             []
142             %Records 
143     end.


--
只为成功找方法,不为失败找理由



--
只为成功找方法,不为失败找理由

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: MapReduce paging result

Parnell Springmeyer
Your email is slightly confusing - I assume by "paging" you mean "pagination"? I recommend not trying to do pagination in Riak.

I can't help you at all with your custom map/reduce phase code in trend_riak.erl without seeing vm.args, trend_riak.erl, &c...

On Aug 9, 2012, at 7:27 PM, 郎咸武 wrote:

Who can answer me?  thanks.

2012/8/10 郎咸武 <[hidden email]>
Hi guys,
    How to do a paging in MapReduce queries?
    I try to write some  method in the module(trend_riak.erl, Add  -pa /home/jason/work/server/trunk/trend_riak    to vm.args).
    I thought it will well, but unfortunately it does not work. Becase the "Reduce" ,the result is wrong.
    Is there the other methods? Thanks.

   Cheers Jason
    
121 get_reduce_bucket_values(Records, {Start, Max}) ->
122     Skip = get_skip_num(Start, Max),
123     io:format("get_reduce_bucket_values Records=~p Skip= ~p~n", [length(Records), Skip]),                                                        
124     if length(Records) >= Skip  ->
125             %%NewList = lists:keysort(#pubsub_item.published, Records),
126             F = fun(#pubsub_item{published=Published1, level=Level1},
127                     #pubsub_item{published=Published2, level=Level2}) ->
128                     if Level1 > Level2 -> true;
129                        Level1 == Level2 ->
130                             if  Published1 > Published2 -> true;
131                                 true -> false
132                             end;
133                        true -> false
134                     end
135                 end,
136             NewList = lists:sort(F, Records),
137             R = lists:sublist(NewList, Skip, Max),
138             io:format("get_reduce_bucket_value R=~p Skip= ~p~n", [length(R), skip]),
139             R;
140        true ->
141             []
142             %Records 
143     end.


--
只为成功找方法,不为失败找理由



--
只为成功找方法,不为失败找理由
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MapReduce paging result

Daniil Churikov
In reply to this post by jason
To do paging in mapreduce queries I make queries based on secondary indexes (2i). My keys is ordered by date and so I could ask date period. If you need results based on not dates but number of keys, you could name your keys like "00001", "00002", etc. But it would be problem with strict number of results if you sometimes drop some keys.
Example of mapred query with 2i(leveldb backend) and riak protocol buffers client:

riakc_pb_socket:mapred(
    Conn,
    {index, <<"page_bucket">>, <<"$key">>, From = "00001", To = "00099"},
    [{map, {modfun, riak_kv_mapreduce, map_object_value}, none, true}]
).