Riak Intermittent Read Failures

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Riak Intermittent Read Failures

markrthomas
Hello

I'm getting intermiitent read failures in my cluster, i.e. timeout

Sometimes an object returns immediately.

Other times, nothing at all and I get a read-timeout.

Any ideas on where I start debugging this issue?

Thanks

Mark
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak Intermittent Read Failures

sean mcevoy
Hi Mark,

I've observed timeouts too but always on serach operation, you might have seen my thread "Solr search response time spikes".

I'm getting stats by polling this every minute:
http://docs.basho.com/riak/kv/2.2.3/developing/api/http/status/

The 99 & 100% response times are most interesting for debugging our problems.
What client & timeout value are you using? I'm using the erlang client where the default timeout is 60 seconds, but I've over ridden that and am using 2 seconds.

Interestingly, over the weekend I've started to see a few put & get timeouts on the application side, but the longest 100% response time is just under a second which points to a network delay.

I'd start by polling these stats and then examining when you get an application side timeout. Maybe check the size stats too, if you can catch which key the operation timed out on it'd be worth checking the object size & sibling count for it. If nothing else this would eliminate the possibility that it's unique to a particular object.

//Sean.


On Sat, Jun 24, 2017 at 12:57 PM, markrthomas <[hidden email]> wrote:
Hello

I'm getting intermiitent read failures in my cluster, i.e. timeout

Sometimes an object returns immediately.

Other times, nothing at all and I get a read-timeout.

Any ideas on where I start debugging this issue?

Thanks

Mark



--
View this message in context: http://riak-users.197444.n3.nabble.com/Riak-Intermittent-Read-Failures-tp4035229.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...