MapReduce queries fail while node is starting

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

MapReduce queries fail while node is starting

Nico Meyer
Hi,

since we upgraded our riak cluster from 0.14 to 1.2, we see MapReduce
queries failing while any of the nodes is starting but not yet ready
(that is while the 'Waiting for service riak_kv'  message still appears
in the logs).
This is quite problematic, since it takes almost 50 minutes for the node
to read the bitcask keydir:
     [info] Wait complete for service riak_kv (3117 seconds)

The following errors appear in the log of the node that handles the
MapReduce query:

2012-11-14 17:49:57.660 [error] emulator Error in process <0.32658.1777>
on node 'riak@172.20.1.11' with exit value:
{{badmatch,{'EXIT',timeout}},[{riak_core_vnode_proxy,call,2,[{file,"src/riak_core_vnode_proxy.erl"},{line,52}]},{riak_pipe_vnode,queue_work_send,4,[{file,"src/riak_pipe_vnode.erl"},{line,331}]},{riak_pipe_vnode,queue_work_erracc...

2012-11-14 17:49:57.798 [error] <0.258.0> Supervisor
riak_pipe_builder_sup had child undefined started with
{riak_pipe_builder,start_link,undefined} at <0.32653.1777> exit with
reason no match of right hand value {'EXIT',timeout} in
riak_core_vnode_proxy:call/2 line 52 in context child_terminated


and this error appears in the log of the node that is still starting up:

[error] Pipe worker startup failed:fitting was gone before startup


Is this a known issue or something we are doing wrong?

Thanks for your help!

Cheers,
Nico


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: MapReduce queries fail while node is starting

bryan-basho
Administrator
On Wed, Nov 14, 2012 at 12:04 PM, Nico Meyer <[hidden email]> wrote:

> The following errors appear in the log of the node that handles the
> MapReduce query:
>
> 2012-11-14 17:49:57.660 [error] emulator Error in process <0.32658.1777> on
> node 'riak@172.20.1.11' with exit value:
> {{badmatch,{'EXIT',timeout}},[{riak_core_vnode_proxy,call,2,[{file,"src/riak_core_vnode_proxy.erl"},{line,52}]},{riak_pipe_vnode,queue_work_send,4,[{file,"src/riak_pipe_vnode.erl"},{line,331}]},{riak_pipe_vnode,queue_work_erracc...
> 2012-11-14 17:49:57.798 [error] <0.258.0> Supervisor riak_pipe_builder_sup
> had child undefined started with {riak_pipe_builder,start_link,undefined} at
> <0.32653.1777> exit with reason no match of right hand value
> {'EXIT',timeout} in riak_core_vnode_proxy:call/2 line 52 in context
> child_terminated
>
> Is this a known issue or something we are doing wrong?

Hi, Nico. I believe this is a known issue:

https://github.com/basho/riak_core/pull/231

It has been fixed on master, and it will be part of the next Riak release.

That fix was also merged into the 1.2 branch, if you're willing and
able to rebuild locally.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com