Errors messages from 2i/range queries, or something else?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Errors messages from 2i/range queries, or something else?

Dave Brady
Our five-node cluster, running 1.4.7, is spewing lots of these messages:

2014-05-30 08:49:20.683 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.8208.361> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,155},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,271310}]
2014-05-30 08:52:30.532 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_1,4}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,252851}]
2014-05-30 08:52:31.123 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_2,5}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,58},{old_heap_size,0},{heap_size,251111}]
2014-05-30 08:52:37.041 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1_1,6}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,190521}]
2014-05-30 08:52:37.149 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1,5}},{message_queue_len,0}] [{timeout,101},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,196361}]
2014-05-30 08:52:37.977 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,123},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,251541}]
2014-05-30 09:02:00.731 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.16282.433> [{initial_call,{erlang,apply,2}},{almost_current_function,{erlang,max,2}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,65},{old_heap_size,0},{heap_size,125642}]

GET and PUT times are shooting into multi-second response.

Can someone give me any hints about where to look, please?

--
Dave Brady


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Errors messages from 2i/range queries, or something else?

Russell Brown
How large is your max_results size for 2i queries, or, if you’re not using pagination, what do you estimate the result size is?

Do you require sorted results? If you don’t, and you’re not using pagination, 1.4.8 might solve your issues since it doesn’t buffer and sort results in memory (see Sorting under http://docs.basho.com/riak/1.4.8/dev/using/2i/#Querying)

Cheers

Russell

On 30 May 2014, at 10:27, Dave Brady <[hidden email]> wrote:

> Our five-node cluster, running 1.4.7, is spewing lots of these messages:
>
> 2014-05-30 08:49:20.683 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.8208.361> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,155},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,271310}]
> 2014-05-30 08:52:30.532 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_1,4}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,252851}]
> 2014-05-30 08:52:31.123 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_2,5}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,58},{old_heap_size,0},{heap_size,251111}]
> 2014-05-30 08:52:37.041 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1_1,6}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,190521}]
> 2014-05-30 08:52:37.149 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1,5}},{message_queue_len,0}] [{timeout,101},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,196361}]
> 2014-05-30 08:52:37.977 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,123},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,251541}]
> 2014-05-30 09:02:00.731 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.16282.433> [{initial_call,{erlang,apply,2}},{almost_current_function,{erlang,max,2}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,65},{old_heap_size,0},{heap_size,125642}]
>
> GET and PUT times are shooting into multi-second response.
>
> Can someone give me any hints about where to look, please?
>
> --
> Dave Brady
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Errors messages from 2i/range queries, or something else?

Dave Brady
Hi Russell,

Thanks for clarifying that it is our 2i queries!

I myself don't know the size of the result sets.  I know pagination is used in some of our programs, though I don't know if all of them use it.

I didn't know that 1.4.8 had those improvements. I'll look into upgrading, too.

Thanks again!

--
Dave Brady

----- Original Message -----
From: "Russell Brown" <[hidden email]>
To: "Dave Brady" <[hidden email]>
Cc: [hidden email]
Sent: Friday, May 30, 2014 11:33:45 AM
Subject: Re: Errors messages from 2i/range queries, or something else?

How large is your max_results size for 2i queries, or, if you’re not using pagination, what do you estimate the result size is?

Do you require sorted results? If you don’t, and you’re not using pagination, 1.4.8 might solve your issues since it doesn’t buffer and sort results in memory (see Sorting under http://docs.basho.com/riak/1.4.8/dev/using/2i/#Querying)

Cheers

Russell

On 30 May 2014, at 10:27, Dave Brady <[hidden email]> wrote:

> Our five-node cluster, running 1.4.7, is spewing lots of these messages:
>
> 2014-05-30 08:49:20.683 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.8208.361> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,155},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,271310}]
> 2014-05-30 08:52:30.532 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_1,4}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,252851}]
> 2014-05-30 08:52:31.123 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_2,5}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,58},{old_heap_size,0},{heap_size,251111}]
> 2014-05-30 08:52:37.041 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1_1,6}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,190521}]
> 2014-05-30 08:52:37.149 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1,5}},{message_queue_len,0}] [{timeout,101},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,196361}]
> 2014-05-30 08:52:37.977 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,123},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,251541}]
> 2014-05-30 09:02:00.731 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.16282.433> [{initial_call,{erlang,apply,2}},{almost_current_function,{erlang,max,2}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,65},{old_heap_size,0},{heap_size,125642}]
>
> GET and PUT times are shooting into multi-second response.
>
> Can someone give me any hints about where to look, please?
>
> --
> Dave Brady
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Errors messages from 2i/range queries, or something else?

Russell Brown

On 30 May 2014, at 11:55, Dave Brady <[hidden email]> wrote:

> Hi Russell,
>
> Thanks for clarifying that it is our 2i queries!

I’m not 100% sure, but it seems possible, those messages look similar to the streaming merge sort code working. Which reminds me, are you streaming the results? It’s way better than buffering all the results in memory before returning them.

>
> I myself don't know the size of the result sets.  I know pagination is used in some of our programs, though I don't know if all of them use it.
>
> I didn't know that 1.4.8 had those improvements. I'll look into upgrading, too.
>
> Thanks again!
>
> --
> Dave Brady
>
> ----- Original Message -----
> From: "Russell Brown" <[hidden email]>
> To: "Dave Brady" <[hidden email]>
> Cc: [hidden email]
> Sent: Friday, May 30, 2014 11:33:45 AM
> Subject: Re: Errors messages from 2i/range queries, or something else?
>
> How large is your max_results size for 2i queries, or, if you’re not using pagination, what do you estimate the result size is?
>
> Do you require sorted results? If you don’t, and you’re not using pagination, 1.4.8 might solve your issues since it doesn’t buffer and sort results in memory (see Sorting under http://docs.basho.com/riak/1.4.8/dev/using/2i/#Querying)
>
> Cheers
>
> Russell
>
> On 30 May 2014, at 10:27, Dave Brady <[hidden email]> wrote:
>
>> Our five-node cluster, running 1.4.7, is spewing lots of these messages:
>>
>> 2014-05-30 08:49:20.683 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.8208.361> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,155},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,271310}]
>> 2014-05-30 08:52:30.532 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_1,4}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,252851}]
>> 2014-05-30 08:52:31.123 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.17361.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge2_2,5}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,58},{old_heap_size,0},{heap_size,251111}]
>> 2014-05-30 08:52:37.041 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1_1,6}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,190521}]
>> 2014-05-30 08:52:37.149 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,split_1,5}},{message_queue_len,0}] [{timeout,101},{old_heap_block_size,0},{heap_block_size,317811},{mbuf_size,0},{stack_size,56},{old_heap_size,0},{heap_size,196361}]
>> 2014-05-30 08:52:37.977 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.29376.375> [{initial_call,{erlang,apply,2}},{almost_current_function,{lists,rmerge3_21_3,6}},{message_queue_len,0}] [{timeout,123},{old_heap_block_size,0},{heap_block_size,514229},{mbuf_size,0},{stack_size,59},{old_heap_size,0},{heap_size,251541}]
>> 2014-05-30 09:02:00.731 [info] <0.83.0>@riak_core_sysmon_handler:handle_event:92 monitor long_gc <0.16282.433> [{initial_call,{erlang,apply,2}},{almost_current_function,{erlang,max,2}},{message_queue_len,0}] [{timeout,103},{old_heap_block_size,0},{heap_block_size,196418},{mbuf_size,0},{stack_size,65},{old_heap_size,0},{heap_size,125642}]
>>
>> GET and PUT times are shooting into multi-second response.
>>
>> Can someone give me any hints about where to look, please?
>>
>> --
>> Dave Brady
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com