error: undef done()

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

error: undef done()

Matthew A. Brown
Hi all,

I'm seeing a strange error raised from an Erlang reduce phase. This
happens consistently with a certain set of inputs but I can't find any
difference between those inputs and myriad other sets of inputs that
produce no error. The error is simply:

[error] error:undef done()

The error message is followed by a dump of the reduce inputs which is,
sadly, truncated before the bit where it would give the call trace.

Our Erlang reduce code does not define or reference a function done(),
but I do note that this function is defined in riak_kv:

https://github.com/basho/riak_kv/blob/master/src/riak_kv_w_reduce.erl#L163

So I'm wondering if this might be a bug in Riak itself? Has anyone ever seen it?

Thanks!
Mat

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: error: undef done()

Matthew A. Brown
Scratch that -- I tracked it down to an actual call to an undefined
function, which was not called "done()" -- I guess we can attribute
this to Erlang's rather inscrutable error messages.

On Tue, Jan 31, 2012 at 13:46, Matthew A. Brown <[hidden email]> wrote:

> Hi all,
>
> I'm seeing a strange error raised from an Erlang reduce phase. This
> happens consistently with a certain set of inputs but I can't find any
> difference between those inputs and myriad other sets of inputs that
> produce no error. The error is simply:
>
> [error] error:undef done()
>
> The error message is followed by a dump of the reduce inputs which is,
> sadly, truncated before the bit where it would give the call trace.
>
> Our Erlang reduce code does not define or reference a function done(),
> but I do note that this function is defined in riak_kv:
>
> https://github.com/basho/riak_kv/blob/master/src/riak_kv_w_reduce.erl#L163
>
> So I'm wondering if this might be a bug in Riak itself? Has anyone ever seen it?
>
> Thanks!
> Mat

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: error: undef done()

bryan-basho
Administrator
In reply to this post by Matthew A. Brown
On Tue, Jan 31, 2012 at 1:46 PM, Matthew A. Brown <[hidden email]> wrote:
> [error] error:undef done()
>
> The error message is followed by a dump of the reduce inputs which is,
> sadly, truncated before the bit where it would give the call trace.
>
> Our Erlang reduce code does not define or reference a function done(),
> but I do note that this function is defined in riak_kv:
>
> https://github.com/basho/riak_kv/blob/master/src/riak_kv_w_reduce.erl#L163

The "done()" you're seeing comes from 6 lines below where you were looking:

https://github.com/basho/riak_kv/blob/master/src/riak_kv_w_reduce.erl#L169

It is intended to give an idea of what stage the reduce fitting was in
when the error happened. Other strings you'll find are "reducing",
"riak_kv_w_reduce init", and "reducing handoff".

I would be interested in knowing what version of Riak you were using
when you encountered this error message.  Improving MapReduce errors
is something we're always working on.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: error: undef done()

Matthew A. Brown
Hi Bryan,

Thanks much for the info. We're running Riak 1.0.3. I think in our
case what would've made this easier to track down would have been not
having the error output cut off in the log -- since the inputs to the
reduce comprised a pretty large set, the output cut off before the
call trace, which appears after the dump of the inputs. It was pretty
easy to track down once I narrowed down the inputs to a manageable
size that still reproduced the error.

Thanks!
Mat

On Wed, Feb 1, 2012 at 10:53, Bryan Fink <[hidden email]> wrote:

> On Tue, Jan 31, 2012 at 1:46 PM, Matthew A. Brown <[hidden email]> wrote:
>> [error] error:undef done()
>>
>> The error message is followed by a dump of the reduce inputs which is,
>> sadly, truncated before the bit where it would give the call trace.
>>
>> Our Erlang reduce code does not define or reference a function done(),
>> but I do note that this function is defined in riak_kv:
>>
>> https://github.com/basho/riak_kv/blob/master/src/riak_kv_w_reduce.erl#L163
>
> The "done()" you're seeing comes from 6 lines below where you were looking:
>
> https://github.com/basho/riak_kv/blob/master/src/riak_kv_w_reduce.erl#L169
>
> It is intended to give an idea of what stage the reduce fitting was in
> when the error happened. Other strings you'll find are "reducing",
> "riak_kv_w_reduce init", and "reducing handoff".
>
> I would be interested in knowing what version of Riak you were using
> when you encountered this error message.  Improving MapReduce errors
> is something we're always working on.
>
> -Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: error: undef done()

bryan-basho
Administrator
On Wed, Feb 1, 2012 at 11:43 AM, Matthew A. Brown <[hidden email]> wrote:
> Thanks much for the info. We're running Riak 1.0.3. I think in our
> case what would've made this easier to track down would have been not
> having the error output cut off in the log -- since the inputs to the
> reduce comprised a pretty large set, the output cut off before the
> call trace, which appears after the dump of the inputs. It was pretty
> easy to track down once I narrowed down the inputs to a manageable
> size that still reproduced the error.

Agreed - it's so hard to know how much context is useful, and how much
is too much.  However, I think you'll find that the upcoming Riak 1.1
release already does much better in this situation.

I just created a small sample to reproduce what you found, where I
have a reduce function that calls bryan:does_not_exist/0, which is
undefined.  In 1.0.3, as you reported, you see no indication of error
in the return value, and useless spew on the error console like:

    13:26:43.720 [error] error:undef done():
       [...so many inputs that the stack trace is omitted...

In the upcoming 1.1 release, using the HTTP interface at /mapred, you
receive a JSON object like:

    {"phase":0,
     "error":"{undef,[{bryan,does_not_exist,[]},{riak_kv_w_reduce,reduce,3},{riak_kv_w_reduce,done,1},{riak_pipe_vnode_worker,wait_for_input,2},{gen_fsm,handle_msg,7},{proc_lib,init_p_do_apply,3}]}"
     "inputs":...}

As well as a much more informational message on the error console:

    13:27:48.143 [error] gen_fsm <0.1484.0> in state wait_for_input
terminated with reason: call to undefined function
bryan:does_not_exist/0 from riak_kv_w_reduce:reduce/3

Hope this helps,
Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: error: undef done()

Matthew A. Brown
Hey Bryan,

That's awesome -- with the Riak 1.1 output figuring out the problem
would've been a snap. Thanks!

Mat

On Wed, Feb 1, 2012 at 13:36, Bryan Fink <[hidden email]> wrote:

> On Wed, Feb 1, 2012 at 11:43 AM, Matthew A. Brown <[hidden email]> wrote:
>> Thanks much for the info. We're running Riak 1.0.3. I think in our
>> case what would've made this easier to track down would have been not
>> having the error output cut off in the log -- since the inputs to the
>> reduce comprised a pretty large set, the output cut off before the
>> call trace, which appears after the dump of the inputs. It was pretty
>> easy to track down once I narrowed down the inputs to a manageable
>> size that still reproduced the error.
>
> Agreed - it's so hard to know how much context is useful, and how much
> is too much.  However, I think you'll find that the upcoming Riak 1.1
> release already does much better in this situation.
>
> I just created a small sample to reproduce what you found, where I
> have a reduce function that calls bryan:does_not_exist/0, which is
> undefined.  In 1.0.3, as you reported, you see no indication of error
> in the return value, and useless spew on the error console like:
>
>    13:26:43.720 [error] error:undef done():
>       [...so many inputs that the stack trace is omitted...
>
> In the upcoming 1.1 release, using the HTTP interface at /mapred, you
> receive a JSON object like:
>
>    {"phase":0,
>     "error":"{undef,[{bryan,does_not_exist,[]},{riak_kv_w_reduce,reduce,3},{riak_kv_w_reduce,done,1},{riak_pipe_vnode_worker,wait_for_input,2},{gen_fsm,handle_msg,7},{proc_lib,init_p_do_apply,3}]}"
>     "inputs":...}
>
> As well as a much more informational message on the error console:
>
>    13:27:48.143 [error] gen_fsm <0.1484.0> in state wait_for_input
> terminated with reason: call to undefined function
> bryan:does_not_exist/0 from riak_kv_w_reduce:reduce/3
>
> Hope this helps,
> Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com