riak errors related to riak_pipe_vnode

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

riak errors related to riak_pipe_vnode

Bhuwan Chawla
I'm seeing tons of errors in our log files and not sure what they mean (Riak 1.2.1):

error.log
<0.314.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup failed:fitting was gone before startup

erlang.log
[error] Pipe worker startup failed:fitting was gone before startup


Any advice is greatly appreciated :-)

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: riak errors related to riak_pipe_vnode

alexmoore
Hi Bhuwan,

That error usually means that you are overloading your cluster in some fashion.  
To fix this you could try a rolling restart of your nodes.

To find the root cause though, can you reply with the following information:

What are the ulimit settings on your riak machines?
Are you running a lot of MapReduce jobs?
Can you clean and attach the logs here?

Thanks,
Alex

On Wed, Apr 17, 2013 at 2:42 PM, Bhuwan Chawla <[hidden email]> wrote:
I'm seeing tons of errors in our log files and not sure what they mean (Riak 1.2.1):

error.log
<0.314.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup failed:fitting was gone before startup

erlang.log
[error] Pipe worker startup failed:fitting was gone before startup


Any advice is greatly appreciated :-)

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: riak errors related to riak_pipe_vnode

Bhuwan Chawla
Thank you very much.  We're going to get Enterprise and will work directly with Basho to better understand our setup.  

Just wanted to reply in case someone in the future comes across the same (or similar) issue.


On Wed, Apr 17, 2013 at 3:09 PM, Alexander Moore <[hidden email]> wrote:
Hi Bhuwan,

That error usually means that you are overloading your cluster in some fashion.  
To fix this you could try a rolling restart of your nodes.

To find the root cause though, can you reply with the following information:

What are the ulimit settings on your riak machines?
Are you running a lot of MapReduce jobs?
Can you clean and attach the logs here?

Thanks,
Alex

On Wed, Apr 17, 2013 at 2:42 PM, Bhuwan Chawla <[hidden email]> wrote:
I'm seeing tons of errors in our log files and not sure what they mean (Riak 1.2.1):

error.log
<0.314.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup failed:fitting was gone before startup

erlang.log
[error] Pipe worker startup failed:fitting was gone before startup


Any advice is greatly appreciated :-)

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: riak errors related to riak_pipe_vnode

bryan-basho
Administrator
Hello. A small update to this issue:

While Alex is correct, that "fitting was gone before startup" messages
in Riak logs can be a symptom of an overloaded cluster, they are most
often nothing more than spam right after an MR query reaches its
timeout. That is why I've just merged a change for the next version of
Riak that lowers that message from 'error' to 'debug' level.

https://github.com/basho/riak_pipe/pull/75

Look forward to less noise in your logs soon.

Cheers,
Bryan


On Tue, Apr 30, 2013 at 8:40 AM, Bhuwan Chawla <[hidden email]> wrote:

> Thank you very much.  We're going to get Enterprise and will work directly
> with Basho to better understand our setup.
>
> Just wanted to reply in case someone in the future comes across the same (or
> similar) issue.
>
>
> On Wed, Apr 17, 2013 at 3:09 PM, Alexander Moore <[hidden email]>
> wrote:
>>
>> Hi Bhuwan,
>>
>> That error usually means that you are overloading your cluster in some
>> fashion.
>> To fix this you could try a rolling restart of your nodes.
>>
>> To find the root cause though, can you reply with the following
>> information:
>>
>> What are the ulimit settings on your riak machines?
>> Are you running a lot of MapReduce jobs?
>> Can you clean and attach the logs here?
>>
>> Thanks,
>> Alex
>>
>> On Wed, Apr 17, 2013 at 2:42 PM, Bhuwan Chawla <[hidden email]> wrote:
>>>
>>> I'm seeing tons of errors in our log files and not sure what they mean
>>> (Riak 1.2.1):
>>>
>>> error.log
>>> <0.314.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
>>> failed:fitting was gone before startup
>>>
>>> erlang.log
>>> [error] Pipe worker startup failed:fitting was gone before startup
>>>
>>>
>>> Any advice is greatly appreciated :-)
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: riak errors related to riak_pipe_vnode

Bhuwan Chawla
Thank you.


On Tue, May 21, 2013 at 3:49 PM, Bryan Fink <[hidden email]> wrote:
Hello. A small update to this issue:

While Alex is correct, that "fitting was gone before startup" messages
in Riak logs can be a symptom of an overloaded cluster, they are most
often nothing more than spam right after an MR query reaches its
timeout. That is why I've just merged a change for the next version of
Riak that lowers that message from 'error' to 'debug' level.

https://github.com/basho/riak_pipe/pull/75

Look forward to less noise in your logs soon.

Cheers,
Bryan


On Tue, Apr 30, 2013 at 8:40 AM, Bhuwan Chawla <[hidden email]> wrote:
> Thank you very much.  We're going to get Enterprise and will work directly
> with Basho to better understand our setup.
>
> Just wanted to reply in case someone in the future comes across the same (or
> similar) issue.
>
>
> On Wed, Apr 17, 2013 at 3:09 PM, Alexander Moore <[hidden email]>
> wrote:
>>
>> Hi Bhuwan,
>>
>> That error usually means that you are overloading your cluster in some
>> fashion.
>> To fix this you could try a rolling restart of your nodes.
>>
>> To find the root cause though, can you reply with the following
>> information:
>>
>> What are the ulimit settings on your riak machines?
>> Are you running a lot of MapReduce jobs?
>> Can you clean and attach the logs here?
>>
>> Thanks,
>> Alex
>>
>> On Wed, Apr 17, 2013 at 2:42 PM, Bhuwan Chawla <[hidden email]> wrote:
>>>
>>> I'm seeing tons of errors in our log files and not sure what they mean
>>> (Riak 1.2.1):
>>>
>>> error.log
>>> <0.314.0>@riak_pipe_vnode:new_worker:766 Pipe worker startup
>>> failed:fitting was gone before startup
>>>
>>> erlang.log
>>> [error] Pipe worker startup failed:fitting was gone before startup
>>>
>>>
>>> Any advice is greatly appreciated :-)
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...