Continuous Crash Report

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Continuous Crash Report

markrthomas

Hello

 

My crash.log is continually filling up with the following message after replacing a node:

 

2017-06-22 08:06:05 =CRASH REPORT====

  crasher:

    initial call: riak_kv_index_hashtree:init/1

    pid: <0.16663.2>

    registered_name: []

    exception exit: {{function_clause,[{riak_core_bucket_props,resolve,['$deleted',[{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,chash_std_keyfun}},{claimant,'[hidden email]'},{dvv_enabled,false},{dw,quorum},{last_write_wins,true},{linkfun,{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},{n_val,3},{name,{<<"commercial">>,<<"commercial_systest">>}},{notfound_ok,false},{old_vclock,86400},{postcommit,[]},{pr,0},{precommit,[]},{pw,0},{r,quorum},{rw,quorum},{search_index,<<"commercial_systest">>},{small_vclock,50},{w,quorum},{young_vclock,20}]],[{file,"src/riak_core_bucket_props.erl"},{line,129}]},{riak_core_metadata_object,'-resolve/2-fun-1-',3,[{file,"src/riak_core_metadata_object.erl"},{line,116}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{dvvset,reconcile,2,[{file,"src/dvvset.erl"},{line,243}]},{riak_core_metadata_object,resolve,2,[{file,"src/riak_core_metadata_object.erl"},{line,118}]},{riak_core_metadata,maybe_resolve,5,[{file,"src/riak_core_metadata.erl"},{line,355}]},{riak_core_metadata,itr_key_values,1,[{file,"src/riak_core_metadata.erl"},{line,226}]},{riak_core_metadata,fold_it,3,[{file,"src/riak_core_metadata.erl"},{line,130}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

    ancestors: [<0.1742.0>,riak_core_vnode_sup,riak_core_sup,<0.240.0>]

    messages: []

    links: []

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 2586

    stack_size: 27

    reductions: 3015

 

And in error.log:

 

2017-06-22 08:07:38.284 [error] <0.3955.4> CRASH REPORT Process <0.3955.4> with 0 neighbours exited with reason: no function clause matching riak_core_bucket_props:resolve('$deleted', [{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,...}},...]) line 129 in gen_server:init_it/6 line 328

 

 

'[hidden email]' is a dead node.

 

Mark Thomas
Technical Lead, UK IT
Equifax Inc.

 

O +44 (0)7908 798 270

[hidden email] 

cid:image005.png@01D2C8D0.9963C110     cid:image006.png@01D2C8D0.9963C110    

 

Equifax Limited is registered in England with Registered No. 2425920. Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax Limited is authorised and regulated by the Financial Conduct Authority.

Equifax Touchstone Limited is registered in Scotland with Registered No. SC113401. Registered Office: Exchange Tower,19 Canning Street, Edinburgh, EH3 8EH.

Equifax Commercial Services Limited is registered in the Republic of Ireland with Registered No. 215393. Registered Office: IDA Business & Technology Park, Rosslare Road, Drinagh, Wexford.

 

This message contains information from Equifax which may be confidential and privileged. If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited. If you have received this transmission in error, please notify by e-mail [hidden email].


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Continuous Crash Report

Johan Sommerfeld
Hi,

I'm not sure but looking at the exception and the code you get function clause error because you have the first argument being '$deleted' and the function guard expects lists as arguments


The reason for it being '$deleted' I don't know, have you done anything to bucket properties or is it just replacing a node?

Regards
Johan Sommerfeld

On 22 June 2017 at 10:11, Mark Richard Thomas <[hidden email]> wrote:

Hello

 

My crash.log is continually filling up with the following message after replacing a node:

 

2017-06-22 08:06:05 =CRASH REPORT====

  crasher:

    initial call: riak_kv_index_hashtree:init/1

    pid: <0.16663.2>

    registered_name: []

    exception exit: {{function_clause,[{riak_core_bucket_props,resolve,['$deleted',[{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,chash_std_keyfun}},{claimant,'[hidden email]'},{dvv_enabled,false},{dw,quorum},{last_write_wins,true},{linkfun,{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},{n_val,3},{name,{<<"commercial">>,<<"commercial_systest">>}},{notfound_ok,false},{old_vclock,86400},{postcommit,[]},{pr,0},{precommit,[]},{pw,0},{r,quorum},{rw,quorum},{search_index,<<"commercial_systest">>},{small_vclock,50},{w,quorum},{young_vclock,20}]],[{file,"src/riak_core_bucket_props.erl"},{line,129}]},{riak_core_metadata_object,'-resolve/2-fun-1-',3,[{file,"src/riak_core_metadata_object.erl"},{line,116}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{dvvset,reconcile,2,[{file,"src/dvvset.erl"},{line,243}]},{riak_core_metadata_object,resolve,2,[{file,"src/riak_core_metadata_object.erl"},{line,118}]},{riak_core_metadata,maybe_resolve,5,[{file,"src/riak_core_metadata.erl"},{line,355}]},{riak_core_metadata,itr_key_values,1,[{file,"src/riak_core_metadata.erl"},{line,226}]},{riak_core_metadata,fold_it,3,[{file,"src/riak_core_metadata.erl"},{line,130}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

    ancestors: [<0.1742.0>,riak_core_vnode_sup,riak_core_sup,<0.240.0>]

    messages: []

    links: []

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 2586

    stack_size: 27

    reductions: 3015

 

And in error.log:

 

2017-06-22 08:07:38.284 [error] <0.3955.4> CRASH REPORT Process <0.3955.4> with 0 neighbours exited with reason: no function clause matching riak_core_bucket_props:resolve('$deleted', [{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,...}},...]) line 129 in gen_server:init_it/6 line 328

 

 

'[hidden email]' is a dead node.

 

Mark Thomas
Technical Lead, UK IT
Equifax Inc.

 

O <a href="tel:+44%207908%20798270" value="+447908798270" target="_blank">+44 (0)7908 798 270

[hidden email] 

cid:image005.png@01D2C8D0.9963C110     cid:image006.png@01D2C8D0.9963C110    

 

Equifax Limited is registered in England with Registered No. 2425920. Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax Limited is authorised and regulated by the Financial Conduct Authority.

Equifax Touchstone Limited is registered in Scotland with Registered No. SC113401. Registered Office: Exchange Tower,19 Canning Street, Edinburgh, EH3 8EH.

Equifax Commercial Services Limited is registered in the Republic of Ireland with Registered No. 215393. Registered Office: IDA Business & Technology Park, Rosslare Road, Drinagh, Wexford.

 

This message contains information from Equifax which may be confidential and privileged. If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited. If you have received this transmission in error, please notify by e-mail [hidden email].


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
Johan Sommerfeld
tel: +46 (0) 70 769 15 73
S2HC Sweden AB
Litsbyvägen 56
187 46 Täby

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

RE: [IE] Re: Continuous Crash Report

markrthomas

Hello

 

force-deleted a node but I made no modification to bucket properties.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Johan Sommerfeld
Sent: 22 June 2017 09:30
To: Mark Richard Thomas
Cc: [hidden email]
Subject: [IE] Re: Continuous Crash Report

 

Hi,

 

I'm not sure but looking at the exception and the code you get function clause error because you have the first argument being '$deleted' and the function guard expects lists as arguments

 

 

The reason for it being '$deleted' I don't know, have you done anything to bucket properties or is it just replacing a node?

 

Regards

Johan Sommerfeld

 

On 22 June 2017 at 10:11, Mark Richard Thomas <[hidden email]> wrote:

Hello

 

My crash.log is continually filling up with the following message after replacing a node:

 

2017-06-22 08:06:05 =CRASH REPORT====

  crasher:

    initial call: riak_kv_index_hashtree:init/1

    pid: <0.16663.2>

    registered_name: []

    exception exit: {{function_clause,[{riak_core_bucket_props,resolve,['$deleted',[{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,chash_std_keyfun}},{claimant,'[hidden email]'},{dvv_enabled,false},{dw,quorum},{last_write_wins,true},{linkfun,{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},{n_val,3},{name,{<<"commercial">>,<<"commercial_systest">>}},{notfound_ok,false},{old_vclock,86400},{postcommit,[]},{pr,0},{precommit,[]},{pw,0},{r,quorum},{rw,quorum},{search_index,<<"commercial_systest">>},{small_vclock,50},{w,quorum},{young_vclock,20}]],[{file,"src/riak_core_bucket_props.erl"},{line,129}]},{riak_core_metadata_object,'-resolve/2-fun-1-',3,[{file,"src/riak_core_metadata_object.erl"},{line,116}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{dvvset,reconcile,2,[{file,"src/dvvset.erl"},{line,243}]},{riak_core_metadata_object,resolve,2,[{file,"src/riak_core_metadata_object.erl"},{line,118}]},{riak_core_metadata,maybe_resolve,5,[{file,"src/riak_core_metadata.erl"},{line,355}]},{riak_core_metadata,itr_key_values,1,[{file,"src/riak_core_metadata.erl"},{line,226}]},{riak_core_metadata,fold_it,3,[{file,"src/riak_core_metadata.erl"},{line,130}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

    ancestors: [<0.1742.0>,riak_core_vnode_sup,riak_core_sup,<0.240.0>]

    messages: []

    links: []

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 2586

    stack_size: 27

    reductions: 3015

 

And in error.log:

 

2017-06-22 08:07:38.284 [error] <0.3955.4> CRASH REPORT Process <0.3955.4> with 0 neighbours exited with reason: no function clause matching riak_core_bucket_props:resolve('$deleted', [{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,...}},...]) line 129 in gen_server:init_it/6 line 328

 

 

'[hidden email]' is a dead node.

 

Mark Thomas
Technical Lead, UK IT
Equifax Inc.

 

O <a href="tel:&#43;44%207908%20798270" target="_blank"> +44 (0)7908 798 270

[hidden email] 

cid:image005.png@01D2C8D0.9963C110     cid:image006.png@01D2C8D0.9963C110    

 

Equifax Limited is registered in England with Registered No. 2425920. Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax Limited is authorised and regulated by the Financial Conduct Authority.

Equifax Touchstone Limited is registered in Scotland with Registered No. SC113401. Registered Office: Exchange Tower,19 Canning Street, Edinburgh, EH3 8EH.

Equifax Commercial Services Limited is registered in the Republic of Ireland with Registered No. 215393. Registered Office: IDA Business & Technology Park, Rosslare Road, Drinagh, Wexford.

 

This message contains information from Equifax which may be confidential and privileged. If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited. If you have received this transmission in error, please notify by e-mail [hidden email].


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



 

--

Johan Sommerfeld
tel: +46 (0) 70 769 15 73
S2HC Sweden AB
Litsbyvägen 56
187 46 Täby

Equifax Limited is registered in England with Registered No. 2425920. Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax Limited is authorised and regulated by the Financial Conduct Authority.

Equifax Touchstone Limited is registered in Scotland with Registered No. SC113401. Registered Office: Exchange Tower,19 Canning Street, Edinburgh, EH3 8EH.

Equifax Commercial Services Limited is registered in the Republic of Ireland with Registered No. 215393. Registered Office: IDA Business & Technology Park, Rosslare Road, Drinagh, Wexford.

 

This message contains information from Equifax which may be confidential and privileged. If you are not an intended recipient, please refrain from any disclosure, copying, distribution or use of this information and note that such actions are prohibited. If you have received this transmission in error, please notify by e-mail [hidden email].


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Stopping solrq supervisor in log; What is this?

Robert Latko
Anybody seen this in their logs?

yz_solrq_sup:stop_queue_pair:130 Stopping solrq supervisor for index <<"search1">>

Searching using the index still works, however I am getting log notices for most my indexes?

If you know, please tell me...  Thank you!

Sincerely,

Robert




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: [IE] Re: Continuous Crash Report

Johan Sommerfeld
In reply to this post by markrthomas
I haven't used bucket types that much so have no real idea of why it
thinks that a bucket type is deleted.

If you read the error log it seem like {name
{<<"commercial">>,<<"commercial_systest">>}}. As I said I'm not that
familiar wit bucket types but having a tuple as name feels odd. What
bucket types do you have and how did they get created.

Regarding the cluster how many nodes are there, what happened to the
node that went away, was it gone for long before being removed. I'm
poking since as far as I can read there is no way of removing a bucket
type so it could be something internal that has gone wrong. How did
you deploy the new node, was it installed using script could it be
that you created the bucket type once more before joining the new
node. did you force-remove and then add a new one or did you do force
replace?

I'm in way over my head with bucket types but maybe someone else can
jump in once they get online.

/J


On 22 June 2017 at 12:31, Mark Richard Thomas <[hidden email]> wrote:

> Hello
>
>
>
> force-deleted a node but I made no modification to bucket properties.
>
>
>
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Johan Sommerfeld
> Sent: 22 June 2017 09:30
> To: Mark Richard Thomas
> Cc: [hidden email]
> Subject: [IE] Re: Continuous Crash Report
>
>
>
> Hi,
>
>
>
> I'm not sure but looking at the exception and the code you get function
> clause error because you have the first argument being '$deleted' and the
> function guard expects lists as arguments
>
>
>
> https://github.com/basho/riak_core/blob/master/src/riak_core_bucket_props.erl#L129
>
>
>
> The reason for it being '$deleted' I don't know, have you done anything to
> bucket properties or is it just replacing a node?
>
>
>
> Regards
>
> Johan Sommerfeld
>
>
>
> On 22 June 2017 at 10:11, Mark Richard Thomas <[hidden email]>
> wrote:
>
> Hello
>
>
>
> My crash.log is continually filling up with the following message after
> replacing a node:
>
>
>
> 2017-06-22 08:06:05 =CRASH REPORT====
>
>   crasher:
>
>     initial call: riak_kv_index_hashtree:init/1
>
>     pid: <0.16663.2>
>
>     registered_name: []
>
>     exception exit:
> {{function_clause,[{riak_core_bucket_props,resolve,['$deleted',[{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,chash_std_keyfun}},{claimant,'[hidden email]'},{dvv_enabled,false},{dw,quorum},{last_write_wins,true},{linkfun,{modfun,riak_kv_wm_link_walker,mapreduce_linkfun}},{n_val,3},{name,{<<"commercial">>,<<"commercial_systest">>}},{notfound_ok,false},{old_vclock,86400},{postcommit,[]},{pr,0},{precommit,[]},{pw,0},{r,quorum},{rw,quorum},{search_index,<<"commercial_systest">>},{small_vclock,50},{w,quorum},{young_vclock,20}]],[{file,"src/riak_core_bucket_props.erl"},{line,129}]},{riak_core_metadata_object,'-resolve/2-fun-1-',3,[{file,"src/riak_core_metadata_object.erl"},{line,116}]},{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},{dvvset,reconcile,2,[{file,"src/dvvset.erl"},{line,243}]},{riak_core_metadata_object,resolve,2,[{file,"src/riak_core_metadata_object.erl"},{line,118}]},{riak_core_metadata,maybe_resolve,5,[{file,"src/riak_core_metadata.erl"},{line,355}]},{riak_core_metadata,itr_key_values,1,[{file,"src/riak_core_metadata.erl"},{line,226}]},{riak_core_metadata,fold_it,3,[{file,"src/riak_core_metadata.erl"},{line,130}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
>
>     ancestors: [<0.1742.0>,riak_core_vnode_sup,riak_core_sup,<0.240.0>]
>
>     messages: []
>
>     links: []
>
>     dictionary: []
>
>     trap_exit: false
>
>     status: running
>
>     heap_size: 2586
>
>     stack_size: 27
>
>     reductions: 3015
>
>
>
> And in error.log:
>
>
>
> 2017-06-22 08:07:38.284 [error] <0.3955.4> CRASH REPORT Process <0.3955.4>
> with 0 neighbours exited with reason: no function clause matching
> riak_core_bucket_props:resolve('$deleted',
> [{active,true},{allow_mult,false},{basic_quorum,true},{big_vclock,50},{chash_keyfun,{riak_core_util,...}},...])
> line 129 in gen_server:init_it/6 line 328
>
>
>
>
>
> '[hidden email]' is a dead node.
>
>
>
> Mark Thomas
> Technical Lead, UK IT
> Equifax Inc.
>
>
>
> O +44 (0)7908 798 270
>
> [hidden email]
>
>
>
>
>
> Equifax Limited is registered in England with Registered No. 2425920.
> Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax
> Limited is authorised and regulated by the Financial Conduct Authority.
>
> Equifax Touchstone Limited is registered in Scotland with Registered No.
> SC113401. Registered Office: Exchange Tower,19 Canning Street, Edinburgh,
> EH3 8EH.
>
> Equifax Commercial Services Limited is registered in the Republic of Ireland
> with Registered No. 215393. Registered Office: IDA Business & Technology
> Park, Rosslare Road, Drinagh, Wexford.
>
>
>
> This message contains information from Equifax which may be confidential and
> privileged. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail [hidden email].
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
>
> --
>
> Johan Sommerfeld
> tel: +46 (0) 70 769 15 73
> S2HC Sweden AB
> Litsbyvägen 56
> 187 46 Täby
>
> Equifax Limited is registered in England with Registered No. 2425920.
> Registered Office: Capital House, 25 Chapel Street, London NW1 5DS. Equifax
> Limited is authorised and regulated by the Financial Conduct Authority.
>
> Equifax Touchstone Limited is registered in Scotland with Registered No.
> SC113401. Registered Office: Exchange Tower,19 Canning Street, Edinburgh,
> EH3 8EH.
>
> Equifax Commercial Services Limited is registered in the Republic of Ireland
> with Registered No. 215393. Registered Office: IDA Business & Technology
> Park, Rosslare Road, Drinagh, Wexford.
>
>
>
> This message contains information from Equifax which may be confidential and
> privileged. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail [hidden email].



--
Johan Sommerfeld
tel: +46 (0) 70 769 15 73
S2HC Sweden AB
Litsbyvägen 56
187 46 Täby

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Stopping solrq supervisor in log; What is this?

Fred Dushin-3
In reply to this post by Robert Latko
This might be a relatively benign message, which might be better at debug scope than info scope.  This would have been introduced in 2.0.8 or 2.2.2 or later.

Essentially, there are solq supervisors for each partition/index pair (for each partition on a node), and when these supervisors start or stop (e.g., when an index is deleted, or when the ring is rebalanced), you should get these log entries.  There may be less common scenarios that cause a solrq supervisor to start/stop (other than, of course, start/stop of the Riak node), but they shouldn't be occurring regularly.

-Fred

On Jun 22, 2017, at 3:12 PM, Robert Latko <[hidden email]> wrote:

Anybody seen this in their logs?

yz_solrq_sup:stop_queue_pair:130 Stopping solrq supervisor for index <<"search1">>

Searching using the index still works, however I am getting log notices for most my indexes?

If you know, please tell me...  Thank you!

Sincerely,

Robert



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Stopping solrq supervisor in log; What is this?

Fred Dushin-2
In reply to this post by Robert Latko
This might be a relatively benign message, which might be better at debug scope than info scope.  This would have been introduced in 2.0.8 or 2.2.2 or later.

Essentially, there are solq supervisors for each partition/index pair (for each partition on a node), and when these supervisors start or stop (e.g., when an index is deleted, or when the ring is rebalanced), you should get these log entries.  There may be less common scenarios that cause a solrq supervisor to start/stop (other than, of course, start/stop of the Riak node), but they shouldn't be occurring regularly.

-Fred

On Jun 22, 2017, at 3:12 PM, Robert Latko <[hidden email]> wrote:

Anybody seen this in their logs?

yz_solrq_sup:stop_queue_pair:130 Stopping solrq supervisor for index <<"search1">>

Searching using the index still works, however I am getting log notices for most my indexes?

If you know, please tell me...  Thank you!

Sincerely,

Robert



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com