Riak Crash

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak Crash

Lucas Cooper
I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz

Any idea what's up? All this crashing is kinda putting me off Riak ._.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Lucas Cooper
It's happened again. The crash and error logs look pretty much the same.

On Sat, Aug 10, 2013 at 8:17 AM, Lucas Cooper <[hidden email]> wrote:
I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz

Any idea what's up? All this crashing is kinda putting me off Riak ._.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Lucas Cooper
And another one... I thought the point of Erlang was that this kind of thing wasn't meant to happen ._.

On Sat, Aug 10, 2013 at 6:12 PM, Lucas Cooper <[hidden email]> wrote:
It's happened again. The crash and error logs look pretty much the same.


On Sat, Aug 10, 2013 at 8:17 AM, Lucas Cooper <[hidden email]> wrote:
I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz

Any idea what's up? All this crashing is kinda putting me off Riak ._.



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Evan Vigil-McClanahan
I won't have time to look at these logs in more detail until later,
but are you sure you're not getting OOM killed?  Usually there is some
sign of what happened when a node went down in console.log as well,
but it just sort of ends.   Check the syslog for OS messages, you
might find something there.

On Sat, Aug 10, 2013 at 6:08 AM, Lucas Cooper <[hidden email]> wrote:

> And another one... I thought the point of Erlang was that this kind of thing
> wasn't meant to happen ._.
>
>
> On Sat, Aug 10, 2013 at 6:12 PM, Lucas Cooper <[hidden email]> wrote:
>>
>> It's happened again. The crash and error logs look pretty much the same.
>>
>>
>> On Sat, Aug 10, 2013 at 8:17 AM, Lucas Cooper <[hidden email]>
>> wrote:
>>>
>>> I had a crash of an entire cluster early this morning, I'm not entirely
>>> sure why, seems to be something with the indexer or Protocol Buffers (or my
>>> use of them). Here are the various logs: http://egf.me/logs.tar.xz
>>>
>>> Any idea what's up? All this crashing is kinda putting me off Riak ._.
>>
>>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Russell Brown
In reply to this post by Lucas Cooper
Are you using riak search?

On 9 Aug 2013, at 18:17, Lucas Cooper <[hidden email]> wrote:

> I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz
>
> Any idea what's up? All this crashing is kinda putting me off Riak ._.
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Lucas Cooper

Yes, Search indexing is enabled.

On 10/08/2013 11:27 PM, "Russell Brown" <[hidden email]> wrote:
Are you using riak search?

On 9 Aug 2013, at 18:17, Lucas Cooper <[hidden email]> wrote:

> I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz
>
> Any idea what's up? All this crashing is kinda putting me off Riak ._.
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Mark Phillips-4
Hey Lucas, 

Like Evan, I haven't had time to dig into the logs. Just so we know what the baseline is here:

* How many nodes of Riak?
* What version of Riak?
* What type of hardware?
* What's your open files limit [1] ?

Mark 






On Sat, Aug 10, 2013 at 9:27 AM, Lucas Cooper <[hidden email]> wrote:

Yes, Search indexing is enabled.

On 10/08/2013 11:27 PM, "Russell Brown" <[hidden email]> wrote:
Are you using riak search?

On 9 Aug 2013, at 18:17, Lucas Cooper <[hidden email]> wrote:

> I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz
>
> Any idea what's up? All this crashing is kinda putting me off Riak ._.
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak Crash

Lucas Cooper
Hey Mark,

As I've said to two of the other Basho guys (privately), I'm running Riak 1.4 on 3 512MB Xen VPS.

I've since come to the conclusion that Riak is not a good solution for me in the short term, it requires a great deal more RAM than I am prepared to give it. Notably, the max_open_files default and minimum value for a vnode is 30, which means that a vnode will always require 120MB of RAM minimum. With a static ring size, you need to have a cluster ready to handle your maximum memory load when you first create it.

This makes Riak rather impractical for me (I don't have the hardware to throw at a Riak cluster right now but I'd like something that can scale) and has led to me switching to MongoDB which, although it requires a little extra configuration in the future, is working great right now. I'm really missing Riak's text search though.

On Sun, Aug 11, 2013 at 3:50 AM, Mark Phillips <[hidden email]> wrote:
Hey Lucas, 

Like Evan, I haven't had time to dig into the logs. Just so we know what the baseline is here:

* How many nodes of Riak?
* What version of Riak?
* What type of hardware?
* What's your open files limit [1] ?

Mark 






On Sat, Aug 10, 2013 at 9:27 AM, Lucas Cooper <[hidden email]> wrote:

Yes, Search indexing is enabled.

On 10/08/2013 11:27 PM, "Russell Brown" <[hidden email]> wrote:
Are you using riak search?

On 9 Aug 2013, at 18:17, Lucas Cooper <[hidden email]> wrote:

> I had a crash of an entire cluster early this morning, I'm not entirely sure why, seems to be something with the indexer or Protocol Buffers (or my use of them). Here are the various logs: http://egf.me/logs.tar.xz
>
> Any idea what's up? All this crashing is kinda putting me off Riak ._.
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com