Very slow acquisition time (99 percentile) while fast median times

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
Hi,

I'm trying to setup a production environment with Riak as backend.
Unfortunately I have very slow write times that bottleneck my whole system.

Here is a sample of one of my node (riak-admin status | grep -e
'^node_put_fsm_time'):
node_put_fsm_time_100 : 3305516
node_put_fsm_time_95 : 230589
node_put_fsm_time_99 : 1694593
node_put_fsm_time_mean : 79864
node_put_fsm_time_median : 14973

As you can see, I have really good times for most of my writes, yet the
mean time is not so good because a few writes are taking long (Up to 3
seconds)
How can I get rid of those slow insert ? Is that intended/normal ?

My setup is the following:
5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,  
0,0 hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
Writes are w=1 dw=0
each host has 32go of ram, that is almost used for system caching only.
My data are stored on an openstack volume that support up to 3000IOPS.

Here is an iostat sample for 1 minute:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           38.00    0.00    1.81    0.03    0.08   60.07

Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
vda               0.37         0.00         2.27          0 136
vdb               9.60         0.00       294.53          0 17672

Thanks,

Guilaume BODDAERT

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Luke Bakken
Hi Guillaume -

What are the "objsize" stats for your cluster?
--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
<[hidden email]> wrote:

> Hi,
>
> I'm trying to setup a production environment with Riak as backend.
> Unfortunately I have very slow write times that bottleneck my whole system.
>
> Here is a sample of one of my node (riak-admin status | grep -e
> '^node_put_fsm_time'):
> node_put_fsm_time_100 : 3305516
> node_put_fsm_time_95 : 230589
> node_put_fsm_time_99 : 1694593
> node_put_fsm_time_mean : 79864
> node_put_fsm_time_median : 14973
>
> As you can see, I have really good times for most of my writes, yet the mean
> time is not so good because a few writes are taking long (Up to 3 seconds)
> How can I get rid of those slow insert ? Is that intended/normal ?
>
> My setup is the following:
> 5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,  0,0
> hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
> Writes are w=1 dw=0
> each host has 32go of ram, that is almost used for system caching only.
> My data are stored on an openstack volume that support up to 3000IOPS.
>
> Here is an iostat sample for 1 minute:
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>           38.00    0.00    1.81    0.03    0.08   60.07
>
> Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
> vda               0.37         0.00         2.27          0 136
> vdb               9.60         0.00       294.53          0 17672
>
> Thanks,
>
> Guilaume BODDAERT
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
Hi Luke,

Well objsize  seems to be 0, that's very troubling.
I can assure you that i am writing 75 items per second at the moment and
that I can pull data from the cluster.


admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'

consistent_get_objsize_100 : 0
consistent_get_objsize_95 : 0
consistent_get_objsize_99 : 0
consistent_get_objsize_mean : 0
consistent_get_objsize_median : 0
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
node_get_fsm_counter_objsize_100 : 0
node_get_fsm_counter_objsize_95 : 0
node_get_fsm_counter_objsize_99 : 0
node_get_fsm_counter_objsize_mean : 0
node_get_fsm_counter_objsize_median : 0
node_get_fsm_map_objsize_100 : 0
node_get_fsm_map_objsize_95 : 0
node_get_fsm_map_objsize_99 : 0
node_get_fsm_map_objsize_mean : 0
node_get_fsm_map_objsize_median : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_set_objsize_100 : 0
node_get_fsm_set_objsize_95 : 0
node_get_fsm_set_objsize_99 : 0
node_get_fsm_set_objsize_mean : 0
node_get_fsm_set_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0



On 02/05/2016 16:13, Luke Bakken wrote:

> Hi Guillaume -
>
> What are the "objsize" stats for your cluster?
> --
> Luke Bakken
> Engineer
> [hidden email]
>
>
> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
> <[hidden email]> wrote:
>> Hi,
>>
>> I'm trying to setup a production environment with Riak as backend.
>> Unfortunately I have very slow write times that bottleneck my whole system.
>>
>> Here is a sample of one of my node (riak-admin status | grep -e
>> '^node_put_fsm_time'):
>> node_put_fsm_time_100 : 3305516
>> node_put_fsm_time_95 : 230589
>> node_put_fsm_time_99 : 1694593
>> node_put_fsm_time_mean : 79864
>> node_put_fsm_time_median : 14973
>>
>> As you can see, I have really good times for most of my writes, yet the mean
>> time is not so good because a few writes are taking long (Up to 3 seconds)
>> How can I get rid of those slow insert ? Is that intended/normal ?
>>
>> My setup is the following:
>> 5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,  0,0
>> hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
>> Writes are w=1 dw=0
>> each host has 32go of ram, that is almost used for system caching only.
>> My data are stored on an openstack volume that support up to 3000IOPS.
>>
>> Here is an iostat sample for 1 minute:
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>            38.00    0.00    1.81    0.03    0.08   60.07
>>
>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
>> vda               0.37         0.00         2.27          0 136
>> vdb               9.60         0.00       294.53          0 17672
>>
>> Thanks,
>>
>> Guilaume BODDAERT
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Luke Bakken
Which Riak client are you using? Do you have it configured to connect
to all nodes in your cluster or just one?
--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert
<[hidden email]> wrote:

> Hi Luke,
>
> Well objsize  seems to be 0, that's very troubling.
> I can assure you that i am writing 75 items per second at the moment and
> that I can pull data from the cluster.
>
>
> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'
>
> consistent_get_objsize_100 : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_objsize_100 : 0
> node_get_fsm_objsize_95 : 0
> node_get_fsm_objsize_99 : 0
> node_get_fsm_objsize_mean : 0
> node_get_fsm_objsize_median : 0
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
>
>
>
> On 02/05/2016 16:13, Luke Bakken wrote:
>>
>> Hi Guillaume -
>>
>> What are the "objsize" stats for your cluster?
>> --
>> Luke Bakken
>> Engineer
>> [hidden email]
>>
>>
>> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
>> <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I'm trying to setup a production environment with Riak as backend.
>>> Unfortunately I have very slow write times that bottleneck my whole
>>> system.
>>>
>>> Here is a sample of one of my node (riak-admin status | grep -e
>>> '^node_put_fsm_time'):
>>> node_put_fsm_time_100 : 3305516
>>> node_put_fsm_time_95 : 230589
>>> node_put_fsm_time_99 : 1694593
>>> node_put_fsm_time_mean : 79864
>>> node_put_fsm_time_median : 14973
>>>
>>> As you can see, I have really good times for most of my writes, yet the
>>> mean
>>> time is not so good because a few writes are taking long (Up to 3
>>> seconds)
>>> How can I get rid of those slow insert ? Is that intended/normal ?
>>>
>>> My setup is the following:
>>> 5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,  0,0
>>> hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
>>> Writes are w=1 dw=0
>>> each host has 32go of ram, that is almost used for system caching only.
>>> My data are stored on an openstack volume that support up to 3000IOPS.
>>>
>>> Here is an iostat sample for 1 minute:
>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>            38.00    0.00    1.81    0.03    0.08   60.07
>>>
>>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
>>> vda               0.37         0.00         2.27          0 136
>>> vdb               9.60         0.00       294.53          0 17672
>>>
>>> Thanks,
>>>
>>> Guilaume BODDAERT
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
My clients are working through an haproxy box configured on round-robin.

I've switched from PBC to HTTP to provide you this:

May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
[02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61
503 222 - - ---- 5/4/2/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
[02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3
57/0/0/54/111 503 222 - - ---- 5/4/1/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454
[02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5
63/0/0/106/169 503 222 - - ---- 5/4/0/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
[02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5
13/0/1/58/72 503 222 - - ---- 5/4/3/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
[02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4
10/0/1/62/73 503 222 - - ---- 5/4/2/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453
[02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1
66/0/0/63/129 503 222 - - ---- 5/4/1/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true
HTTP/1.1"

Yet objsize is still 0 on puts:
admin@riak1:~$ sudo riak-admin status | grep -e 'objsize'
consistent_get_objsize_100 : 0
consistent_get_objsize_95 : 0
consistent_get_objsize_99 : 0
consistent_get_objsize_mean : 0
consistent_get_objsize_median : 0
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
node_get_fsm_counter_objsize_100 : 0
node_get_fsm_counter_objsize_95 : 0
node_get_fsm_counter_objsize_99 : 0
node_get_fsm_counter_objsize_mean : 0
node_get_fsm_counter_objsize_median : 0
node_get_fsm_map_objsize_100 : 0
node_get_fsm_map_objsize_95 : 0
node_get_fsm_map_objsize_99 : 0
node_get_fsm_map_objsize_mean : 0
node_get_fsm_map_objsize_median : 0
node_get_fsm_objsize_100 : 10916
node_get_fsm_objsize_95 : 7393
node_get_fsm_objsize_99 : 8845
node_get_fsm_objsize_mean : 4098
node_get_fsm_objsize_median : 3891
node_get_fsm_set_objsize_100 : 0
node_get_fsm_set_objsize_95 : 0
node_get_fsm_set_objsize_99 : 0
node_get_fsm_set_objsize_mean : 0
node_get_fsm_set_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0



G.

On 02/05/2016 17:21, Luke Bakken wrote:

> Which Riak client are you using? Do you have it configured to connect
> to all nodes in your cluster or just one?
> --
> Luke Bakken
> Engineer
> [hidden email]
>
>
> On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert
> <[hidden email]> wrote:
>> Hi Luke,
>>
>> Well objsize  seems to be 0, that's very troubling.
>> I can assure you that i am writing 75 items per second at the moment and
>> that I can pull data from the cluster.
>>
>>
>> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'
>>
>> consistent_get_objsize_100 : 0
>> consistent_get_objsize_95 : 0
>> consistent_get_objsize_99 : 0
>> consistent_get_objsize_mean : 0
>> consistent_get_objsize_median : 0
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> node_get_fsm_counter_objsize_100 : 0
>> node_get_fsm_counter_objsize_95 : 0
>> node_get_fsm_counter_objsize_99 : 0
>> node_get_fsm_counter_objsize_mean : 0
>> node_get_fsm_counter_objsize_median : 0
>> node_get_fsm_map_objsize_100 : 0
>> node_get_fsm_map_objsize_95 : 0
>> node_get_fsm_map_objsize_99 : 0
>> node_get_fsm_map_objsize_mean : 0
>> node_get_fsm_map_objsize_median : 0
>> node_get_fsm_objsize_100 : 0
>> node_get_fsm_objsize_95 : 0
>> node_get_fsm_objsize_99 : 0
>> node_get_fsm_objsize_mean : 0
>> node_get_fsm_objsize_median : 0
>> node_get_fsm_set_objsize_100 : 0
>> node_get_fsm_set_objsize_95 : 0
>> node_get_fsm_set_objsize_99 : 0
>> node_get_fsm_set_objsize_mean : 0
>> node_get_fsm_set_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>>
>>
>>
>> On 02/05/2016 16:13, Luke Bakken wrote:
>>> Hi Guillaume -
>>>
>>> What are the "objsize" stats for your cluster?
>>> --
>>> Luke Bakken
>>> Engineer
>>> [hidden email]
>>>
>>>
>>> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
>>> <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> I'm trying to setup a production environment with Riak as backend.
>>>> Unfortunately I have very slow write times that bottleneck my whole
>>>> system.
>>>>
>>>> Here is a sample of one of my node (riak-admin status | grep -e
>>>> '^node_put_fsm_time'):
>>>> node_put_fsm_time_100 : 3305516
>>>> node_put_fsm_time_95 : 230589
>>>> node_put_fsm_time_99 : 1694593
>>>> node_put_fsm_time_mean : 79864
>>>> node_put_fsm_time_median : 14973
>>>>
>>>> As you can see, I have really good times for most of my writes, yet the
>>>> mean
>>>> time is not so good because a few writes are taking long (Up to 3
>>>> seconds)
>>>> How can I get rid of those slow insert ? Is that intended/normal ?
>>>>
>>>> My setup is the following:
>>>> 5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,  0,0
>>>> hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
>>>> Writes are w=1 dw=0
>>>> each host has 32go of ram, that is almost used for system caching only.
>>>> My data are stored on an openstack volume that support up to 3000IOPS.
>>>>
>>>> Here is an iostat sample for 1 minute:
>>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>>             38.00    0.00    1.81    0.03    0.08   60.07
>>>>
>>>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
>>>> vda               0.37         0.00         2.27          0 136
>>>> vdb               9.60         0.00       294.53          0 17672
>>>>
>>>> Thanks,
>>>>
>>>> Guilaume BODDAERT
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> [hidden email]
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Luke Bakken
Could you please check the objsize stats on every Riak node? If they
are all zero then ... ????
--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert
<[hidden email]> wrote:

> My clients are working through an haproxy box configured on round-robin.
>
> I've switched from PBC to HTTP to provide you this:
>
> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
> [02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 503
> 222 - - ---- 5/4/2/1/0 0/0 "PUT
> /types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true
> HTTP/1.1"
> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
> [02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 57/0/0/54/111
> 503 222 - - ---- 5/4/1/1/0 0/0 "PUT
> /types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true
> HTTP/1.1"
> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454
> [02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 63/0/0/106/169
> 503 222 - - ---- 5/4/0/1/0 0/0 "PUT
> /types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true
> HTTP/1.1"
> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
> [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5 13/0/1/58/72
> 503 222 - - ---- 5/4/3/1/0 0/0 "PUT
> /types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true
> HTTP/1.1"
> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
> [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4 10/0/1/62/73
> 503 222 - - ---- 5/4/2/1/0 0/0 "PUT
> /types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true
> HTTP/1.1"
> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453
> [02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1 66/0/0/63/129
> 503 222 - - ---- 5/4/1/1/0 0/0 "PUT
> /types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true
> HTTP/1.1"
>
> Yet objsize is still 0 on puts:
> admin@riak1:~$ sudo riak-admin status | grep -e 'objsize'
> consistent_get_objsize_100 : 0
> consistent_get_objsize_95 : 0
> consistent_get_objsize_99 : 0
> consistent_get_objsize_mean : 0
> consistent_get_objsize_median : 0
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> node_get_fsm_counter_objsize_100 : 0
> node_get_fsm_counter_objsize_95 : 0
> node_get_fsm_counter_objsize_99 : 0
> node_get_fsm_counter_objsize_mean : 0
> node_get_fsm_counter_objsize_median : 0
> node_get_fsm_map_objsize_100 : 0
> node_get_fsm_map_objsize_95 : 0
> node_get_fsm_map_objsize_99 : 0
> node_get_fsm_map_objsize_mean : 0
> node_get_fsm_map_objsize_median : 0
> node_get_fsm_objsize_100 : 10916
> node_get_fsm_objsize_95 : 7393
> node_get_fsm_objsize_99 : 8845
> node_get_fsm_objsize_mean : 4098
> node_get_fsm_objsize_median : 3891
> node_get_fsm_set_objsize_100 : 0
> node_get_fsm_set_objsize_95 : 0
> node_get_fsm_set_objsize_99 : 0
> node_get_fsm_set_objsize_mean : 0
> node_get_fsm_set_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
>
>
> G.
>
>
> On 02/05/2016 17:21, Luke Bakken wrote:
>>
>> Which Riak client are you using? Do you have it configured to connect
>> to all nodes in your cluster or just one?
>> --
>> Luke Bakken
>> Engineer
>> [hidden email]
>>
>>
>> On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert
>> <[hidden email]> wrote:
>>>
>>> Hi Luke,
>>>
>>> Well objsize  seems to be 0, that's very troubling.
>>> I can assure you that i am writing 75 items per second at the moment and
>>> that I can pull data from the cluster.
>>>
>>>
>>> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'
>>>
>>> consistent_get_objsize_100 : 0
>>> consistent_get_objsize_95 : 0
>>> consistent_get_objsize_99 : 0
>>> consistent_get_objsize_mean : 0
>>> consistent_get_objsize_median : 0
>>> consistent_put_objsize_100 : 0
>>> consistent_put_objsize_95 : 0
>>> consistent_put_objsize_99 : 0
>>> consistent_put_objsize_mean : 0
>>> consistent_put_objsize_median : 0
>>> node_get_fsm_counter_objsize_100 : 0
>>> node_get_fsm_counter_objsize_95 : 0
>>> node_get_fsm_counter_objsize_99 : 0
>>> node_get_fsm_counter_objsize_mean : 0
>>> node_get_fsm_counter_objsize_median : 0
>>> node_get_fsm_map_objsize_100 : 0
>>> node_get_fsm_map_objsize_95 : 0
>>> node_get_fsm_map_objsize_99 : 0
>>> node_get_fsm_map_objsize_mean : 0
>>> node_get_fsm_map_objsize_median : 0
>>> node_get_fsm_objsize_100 : 0
>>> node_get_fsm_objsize_95 : 0
>>> node_get_fsm_objsize_99 : 0
>>> node_get_fsm_objsize_mean : 0
>>> node_get_fsm_objsize_median : 0
>>> node_get_fsm_set_objsize_100 : 0
>>> node_get_fsm_set_objsize_95 : 0
>>> node_get_fsm_set_objsize_99 : 0
>>> node_get_fsm_set_objsize_mean : 0
>>> node_get_fsm_set_objsize_median : 0
>>> write_once_put_objsize_100 : 0
>>> write_once_put_objsize_95 : 0
>>> write_once_put_objsize_99 : 0
>>> write_once_put_objsize_mean : 0
>>> write_once_put_objsize_median : 0
>>>
>>>
>>>
>>>
>>> On 02/05/2016 16:13, Luke Bakken wrote:
>>>>
>>>> Hi Guillaume -
>>>>
>>>> What are the "objsize" stats for your cluster?
>>>> --
>>>> Luke Bakken
>>>> Engineer
>>>> [hidden email]
>>>>
>>>>
>>>> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
>>>> <[hidden email]> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm trying to setup a production environment with Riak as backend.
>>>>> Unfortunately I have very slow write times that bottleneck my whole
>>>>> system.
>>>>>
>>>>> Here is a sample of one of my node (riak-admin status | grep -e
>>>>> '^node_put_fsm_time'):
>>>>> node_put_fsm_time_100 : 3305516
>>>>> node_put_fsm_time_95 : 230589
>>>>> node_put_fsm_time_99 : 1694593
>>>>> node_put_fsm_time_mean : 79864
>>>>> node_put_fsm_time_median : 14973
>>>>>
>>>>> As you can see, I have really good times for most of my writes, yet the
>>>>> mean
>>>>> time is not so good because a few writes are taking long (Up to 3
>>>>> seconds)
>>>>> How can I get rid of those slow insert ? Is that intended/normal ?
>>>>>
>>>>> My setup is the following:
>>>>> 5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,
>>>>> 0,0
>>>>> hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
>>>>> Writes are w=1 dw=0
>>>>> each host has 32go of ram, that is almost used for system caching only.
>>>>> My data are stored on an openstack volume that support up to 3000IOPS.
>>>>>
>>>>> Here is an iostat sample for 1 minute:
>>>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>>>             38.00    0.00    1.81    0.03    0.08   60.07
>>>>>
>>>>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
>>>>> vda               0.37         0.00         2.27          0 136
>>>>> vdb               9.60         0.00       294.53          0 17672
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Guilaume BODDAERT
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [hidden email]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
Here we go for a complete round of my hosts, all are objsize : 0

Here is a sample answer (headers only, that are followed by the full set
of JSON content) from the RIAK5 host

HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Link: </buckets/twitter>; rel="up"
Last-Modified: Mon, 02 May 2016 15:40:20 GMT
ETag: "2l2QODpewyBZQFqDnyEy3F"
Date: Mon, 02 May 2016 15:40:20 GMT
Content-Type: application/json
Content-Length: 10722

Below the riak-admin status output.


admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'

consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'

consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

On 02/05/2016 17:32, Luke Bakken wrote:

> Could you please check the objsize stats on every Riak node? If they
> are all zero then ... ????
> --
> Luke Bakken
> Engineer
> [hidden email]
>
>
> On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert
> <[hidden email]> wrote:
>> My clients are working through an haproxy box configured on round-robin.
>>
>> I've switched from PBC to HTTP to provide you this:
>>
>> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
>> [02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 503
>> 222 - - ---- 5/4/2/1/0 0/0 "PUT
>> /types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true
>> HTTP/1.1"
>> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
>> [02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 57/0/0/54/111
>> 503 222 - - ---- 5/4/1/1/0 0/0 "PUT
>> /types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true
>> HTTP/1.1"
>> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454
>> [02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 63/0/0/106/169
>> 503 222 - - ---- 5/4/0/1/0 0/0 "PUT
>> /types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true
>> HTTP/1.1"
>> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
>> [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5 13/0/1/58/72
>> 503 222 - - ---- 5/4/3/1/0 0/0 "PUT
>> /types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true
>> HTTP/1.1"
>> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
>> [02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4 10/0/1/62/73
>> 503 222 - - ---- 5/4/2/1/0 0/0 "PUT
>> /types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true
>> HTTP/1.1"
>> May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453
>> [02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1 66/0/0/63/129
>> 503 222 - - ---- 5/4/1/1/0 0/0 "PUT
>> /types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true
>> HTTP/1.1"
>>
>> Yet objsize is still 0 on puts:
>> admin@riak1:~$ sudo riak-admin status | grep -e 'objsize'
>> consistent_get_objsize_100 : 0
>> consistent_get_objsize_95 : 0
>> consistent_get_objsize_99 : 0
>> consistent_get_objsize_mean : 0
>> consistent_get_objsize_median : 0
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> node_get_fsm_counter_objsize_100 : 0
>> node_get_fsm_counter_objsize_95 : 0
>> node_get_fsm_counter_objsize_99 : 0
>> node_get_fsm_counter_objsize_mean : 0
>> node_get_fsm_counter_objsize_median : 0
>> node_get_fsm_map_objsize_100 : 0
>> node_get_fsm_map_objsize_95 : 0
>> node_get_fsm_map_objsize_99 : 0
>> node_get_fsm_map_objsize_mean : 0
>> node_get_fsm_map_objsize_median : 0
>> node_get_fsm_objsize_100 : 10916
>> node_get_fsm_objsize_95 : 7393
>> node_get_fsm_objsize_99 : 8845
>> node_get_fsm_objsize_mean : 4098
>> node_get_fsm_objsize_median : 3891
>> node_get_fsm_set_objsize_100 : 0
>> node_get_fsm_set_objsize_95 : 0
>> node_get_fsm_set_objsize_99 : 0
>> node_get_fsm_set_objsize_mean : 0
>> node_get_fsm_set_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>>
>>
>> G.
>>
>>
>> On 02/05/2016 17:21, Luke Bakken wrote:
>>> Which Riak client are you using? Do you have it configured to connect
>>> to all nodes in your cluster or just one?
>>> --
>>> Luke Bakken
>>> Engineer
>>> [hidden email]
>>>
>>>
>>> On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert
>>> <[hidden email]> wrote:
>>>> Hi Luke,
>>>>
>>>> Well objsize  seems to be 0, that's very troubling.
>>>> I can assure you that i am writing 75 items per second at the moment and
>>>> that I can pull data from the cluster.
>>>>
>>>>
>>>> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'
>>>>
>>>> consistent_get_objsize_100 : 0
>>>> consistent_get_objsize_95 : 0
>>>> consistent_get_objsize_99 : 0
>>>> consistent_get_objsize_mean : 0
>>>> consistent_get_objsize_median : 0
>>>> consistent_put_objsize_100 : 0
>>>> consistent_put_objsize_95 : 0
>>>> consistent_put_objsize_99 : 0
>>>> consistent_put_objsize_mean : 0
>>>> consistent_put_objsize_median : 0
>>>> node_get_fsm_counter_objsize_100 : 0
>>>> node_get_fsm_counter_objsize_95 : 0
>>>> node_get_fsm_counter_objsize_99 : 0
>>>> node_get_fsm_counter_objsize_mean : 0
>>>> node_get_fsm_counter_objsize_median : 0
>>>> node_get_fsm_map_objsize_100 : 0
>>>> node_get_fsm_map_objsize_95 : 0
>>>> node_get_fsm_map_objsize_99 : 0
>>>> node_get_fsm_map_objsize_mean : 0
>>>> node_get_fsm_map_objsize_median : 0
>>>> node_get_fsm_objsize_100 : 0
>>>> node_get_fsm_objsize_95 : 0
>>>> node_get_fsm_objsize_99 : 0
>>>> node_get_fsm_objsize_mean : 0
>>>> node_get_fsm_objsize_median : 0
>>>> node_get_fsm_set_objsize_100 : 0
>>>> node_get_fsm_set_objsize_95 : 0
>>>> node_get_fsm_set_objsize_99 : 0
>>>> node_get_fsm_set_objsize_mean : 0
>>>> node_get_fsm_set_objsize_median : 0
>>>> write_once_put_objsize_100 : 0
>>>> write_once_put_objsize_95 : 0
>>>> write_once_put_objsize_99 : 0
>>>> write_once_put_objsize_mean : 0
>>>> write_once_put_objsize_median : 0
>>>>
>>>>
>>>>
>>>>
>>>> On 02/05/2016 16:13, Luke Bakken wrote:
>>>>> Hi Guillaume -
>>>>>
>>>>> What are the "objsize" stats for your cluster?
>>>>> --
>>>>> Luke Bakken
>>>>> Engineer
>>>>> [hidden email]
>>>>>
>>>>>
>>>>> On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
>>>>> <[hidden email]> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm trying to setup a production environment with Riak as backend.
>>>>>> Unfortunately I have very slow write times that bottleneck my whole
>>>>>> system.
>>>>>>
>>>>>> Here is a sample of one of my node (riak-admin status | grep -e
>>>>>> '^node_put_fsm_time'):
>>>>>> node_put_fsm_time_100 : 3305516
>>>>>> node_put_fsm_time_95 : 230589
>>>>>> node_put_fsm_time_99 : 1694593
>>>>>> node_put_fsm_time_mean : 79864
>>>>>> node_put_fsm_time_median : 14973
>>>>>>
>>>>>> As you can see, I have really good times for most of my writes, yet the
>>>>>> mean
>>>>>> time is not so good because a few writes are taking long (Up to 3
>>>>>> seconds)
>>>>>> How can I get rid of those slow insert ? Is that intended/normal ?
>>>>>>
>>>>>> My setup is the following:
>>>>>> 5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,
>>>>>> 0,0
>>>>>> hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
>>>>>> Writes are w=1 dw=0
>>>>>> each host has 32go of ram, that is almost used for system caching only.
>>>>>> My data are stored on an openstack volume that support up to 3000IOPS.
>>>>>>
>>>>>> Here is an iostat sample for 1 minute:
>>>>>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>>>>>              38.00    0.00    1.81    0.03    0.08   60.07
>>>>>>
>>>>>> Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
>>>>>> vda               0.37         0.00         2.27          0 136
>>>>>> vdb               9.60         0.00       294.53          0 17672
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Guilaume BODDAERT
>>>>>>
>>>>>> _______________________________________________
>>>>>> riak-users mailing list
>>>>>> [hidden email]
>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> [hidden email]
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Alexander Sicular-2
I believe you should be looking for the get_fsm_objsize stats listed here: http://docs.basho.com/riak/kv/2.1.4/using/cluster-operations/inspecting-node/#get-fsm-objsize . Unless you are using consistent bucket types or write once bucket types.

-Alexander



Alexander Sicular
Solutions Architect
Basho Technologies
9175130679
@siculars

On Mon, May 2, 2016 at 10:43 AM, Guillaume Boddaert <[hidden email]> wrote:
Here we go for a complete round of my hosts, all are objsize : 0

Here is a sample answer (headers only, that are followed by the full set of JSON content) from the RIAK5 host

HTTP/1.1 200 OK
X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA
Vary: Accept-Encoding
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Link: </buckets/twitter>; rel="up"
Last-Modified: Mon, 02 May 2016 15:40:20 GMT
ETag: "2l2QODpewyBZQFqDnyEy3F"
Date: Mon, 02 May 2016 15:40:20 GMT
Content-Type: application/json
Content-Length: 10722

Below the riak-admin status output.


admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'

consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'

consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0

On 02/05/2016 17:32, Luke Bakken wrote:
Could you please check the objsize stats on every Riak node? If they
are all zero then ... ????
--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 8:26 AM, Guillaume Boddaert
<[hidden email]> wrote:
My clients are working through an haproxy box configured on round-robin.

I've switched from PBC to HTTP to provide you this:

May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
[02/May/2016:15:24:12.390] riak_rest riak_rest_backend/riak2 6/0/1/54/61 503
222 - - ---- 5/4/2/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126697699594240?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
[02/May/2016:15:24:12.339] riak_rest riak_rest_backend/riak3 57/0/0/54/111
503 222 - - ---- 5/4/1/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698291023872?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53454
[02/May/2016:15:24:12.285] riak_rest riak_rest_backend/riak5 63/0/0/106/169
503 222 - - ---- 5/4/0/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698249064449?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53455
[02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak5 13/0/1/58/72
503 222 - - ---- 5/4/3/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126696256802816?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53456
[02/May/2016:15:24:12.451] riak_rest riak_rest_backend/riak4 10/0/1/62/73
503 222 - - ---- 5/4/2/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698425212929?dw=1&timeout=50&returnbody=true
HTTP/1.1"
May  2 15:24:12 intrabalancer haproxy[29677]: my_daemon_box:53453
[02/May/2016:15:24:12.397] riak_rest riak_rest_backend/riak1 66/0/0/63/129
503 222 - - ---- 5/4/1/1/0 0/0 "PUT
/types/activity_fr/buckets/twitter/keys/727126698215485440?dw=1&timeout=50&returnbody=true
HTTP/1.1"

Yet objsize is still 0 on puts:
admin@riak1:~$ sudo riak-admin status | grep -e 'objsize'
consistent_get_objsize_100 : 0
consistent_get_objsize_95 : 0
consistent_get_objsize_99 : 0
consistent_get_objsize_mean : 0
consistent_get_objsize_median : 0
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
node_get_fsm_counter_objsize_100 : 0
node_get_fsm_counter_objsize_95 : 0
node_get_fsm_counter_objsize_99 : 0
node_get_fsm_counter_objsize_mean : 0
node_get_fsm_counter_objsize_median : 0
node_get_fsm_map_objsize_100 : 0
node_get_fsm_map_objsize_95 : 0
node_get_fsm_map_objsize_99 : 0
node_get_fsm_map_objsize_mean : 0
node_get_fsm_map_objsize_median : 0
node_get_fsm_objsize_100 : 10916
node_get_fsm_objsize_95 : 7393
node_get_fsm_objsize_99 : 8845
node_get_fsm_objsize_mean : 4098
node_get_fsm_objsize_median : 3891
node_get_fsm_set_objsize_100 : 0
node_get_fsm_set_objsize_95 : 0
node_get_fsm_set_objsize_99 : 0
node_get_fsm_set_objsize_mean : 0
node_get_fsm_set_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0



G.


On 02/05/2016 17:21, Luke Bakken wrote:
Which Riak client are you using? Do you have it configured to connect
to all nodes in your cluster or just one?
--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 7:40 AM, Guillaume Boddaert
<[hidden email]> wrote:
Hi Luke,

Well objsize  seems to be 0, that's very troubling.
I can assure you that i am writing 75 items per second at the moment and
that I can pull data from the cluster.


admin@riak3:~$ sudo riak-admin status | grep -e 'objsize'

consistent_get_objsize_100 : 0
consistent_get_objsize_95 : 0
consistent_get_objsize_99 : 0
consistent_get_objsize_mean : 0
consistent_get_objsize_median : 0
consistent_put_objsize_100 : 0
consistent_put_objsize_95 : 0
consistent_put_objsize_99 : 0
consistent_put_objsize_mean : 0
consistent_put_objsize_median : 0
node_get_fsm_counter_objsize_100 : 0
node_get_fsm_counter_objsize_95 : 0
node_get_fsm_counter_objsize_99 : 0
node_get_fsm_counter_objsize_mean : 0
node_get_fsm_counter_objsize_median : 0
node_get_fsm_map_objsize_100 : 0
node_get_fsm_map_objsize_95 : 0
node_get_fsm_map_objsize_99 : 0
node_get_fsm_map_objsize_mean : 0
node_get_fsm_map_objsize_median : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_set_objsize_100 : 0
node_get_fsm_set_objsize_95 : 0
node_get_fsm_set_objsize_99 : 0
node_get_fsm_set_objsize_mean : 0
node_get_fsm_set_objsize_median : 0
write_once_put_objsize_100 : 0
write_once_put_objsize_95 : 0
write_once_put_objsize_99 : 0
write_once_put_objsize_mean : 0
write_once_put_objsize_median : 0




On 02/05/2016 16:13, Luke Bakken wrote:
Hi Guillaume -

What are the "objsize" stats for your cluster?
--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 4:45 AM, Guillaume Boddaert
<[hidden email]> wrote:
Hi,

I'm trying to setup a production environment with Riak as backend.
Unfortunately I have very slow write times that bottleneck my whole
system.

Here is a sample of one of my node (riak-admin status | grep -e
'^node_put_fsm_time'):
node_put_fsm_time_100 : 3305516
node_put_fsm_time_95 : 230589
node_put_fsm_time_99 : 1694593
node_put_fsm_time_mean : 79864
node_put_fsm_time_median : 14973

As you can see, I have really good times for most of my writes, yet the
mean
time is not so good because a few writes are taking long (Up to 3
seconds)
How can I get rid of those slow insert ? Is that intended/normal ?

My setup is the following:
5 hosts (2CPU, %Cpu(s): 47,1 us,  1,3 sy,  0,0 ni, 51,3 id,  0,0 wa,
0,0
hi,  0,2 si,  0,0 st) , ring_size: 128, aae disabled.
Writes are w=1 dw=0
each host has 32go of ram, that is almost used for system caching only.
My data are stored on an openstack volume that support up to 3000IOPS.

Here is an iostat sample for 1 minute:
avg-cpu:  %user   %nice %system %iowait  %steal   %idle
             38.00    0.00    1.81    0.03    0.08   60.07

Device:            tps    kB_read/s    kB_wrtn/s    kB_read kB_wrtn
vda               0.37         0.00         2.27          0 136
vdb               9.60         0.00       294.53          0 17672

Thanks,

Guilaume BODDAERT

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Luke Bakken
In reply to this post by Guillaume Boddaert
Guillaume -

Some colleagues had me carefully re-read those stats. You'll notice
that those "put" stats are only for consistent or write_once
operations, so they don't apply to you.

Your read stats show objects well within Riak's recommended object size:

node_get_fsm_objsize_100 : 10916
node_get_fsm_objsize_95 : 7393
node_get_fsm_objsize_99 : 8845
node_get_fsm_objsize_mean : 4098
node_get_fsm_objsize_median : 3891

So that is not the issue.

Are you using Bitcask? If so, please apply these sysctl settings:

http://docs.basho.com/riak/kv/2.1.4/using/performance/#optional-i-o-settings

If you are using the default "vm.dirty_*" settings Linux will appear
to pause as it flushes disk buffers to the underlying device. The
settings in the document change this so that flushes happen more often
and asynchronously.

--
Luke Bakken
Engineer
[hidden email]


On Mon, May 2, 2016 at 8:43 AM, Guillaume Boddaert
<[hidden email]> wrote:

> Here we go for a complete round of my hosts, all are objsize : 0
>
> Here is a sample answer (headers only, that are followed by the full set of
> JSON content) from the RIAK5 host
>
> HTTP/1.1 200 OK
> X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA
> Vary: Accept-Encoding
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Link: </buckets/twitter>; rel="up"
> Last-Modified: Mon, 02 May 2016 15:40:20 GMT
> ETag: "2l2QODpewyBZQFqDnyEy3F"
> Date: Mon, 02 May 2016 15:40:20 GMT
> Content-Type: application/json
> Content-Length: 10722
>
> Below the riak-admin status output.
>
>
> admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
> admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
> admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
> admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>
> consistent_put_objsize_100 : 0
> consistent_put_objsize_95 : 0
> consistent_put_objsize_99 : 0
> consistent_put_objsize_mean : 0
> consistent_put_objsize_median : 0
> write_once_put_objsize_100 : 0
> write_once_put_objsize_95 : 0
> write_once_put_objsize_99 : 0
> write_once_put_objsize_mean : 0
> write_once_put_objsize_median : 0
>
> On 02/05/2016 17:32, Luke Bakken wrote:
>>
>> Could you please check the objsize stats on every Riak node? If they
>> are all zero then ... ????
>> --
>> Luke Bakken
>> Engineer
>> [hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if
the problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't
rely anymore on loadbalancer and also corrected some minor performance
issue on my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

 From my application server perspective, 23% of the calls are rejected
by timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s.
That's better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
     "props": {
         "active": true,
         "allow_mult": false,
         "basic_quorum": false,
         "big_vclock": 50,
         "chash_keyfun": {
             "fun": "chash_std_keyfun",
             "mod": "riak_core_util"
         },
         "claimant": "[hidden email]",
         "dvv_enabled": false,
         "dw": "quorum",
         "last_write_wins": true,
         "linkfun": {
             "fun": "mapreduce_linkfun",
             "mod": "riak_kv_wm_link_walker"
         },
         "n_val": 3,
         "notfound_ok": true,
         "old_vclock": 86400,
         "postcommit": [],
         "pr": 0,
         "precommit": [],
         "pw": 0,
         "r": "quorum",
         "rw": "quorum",
         "search_index": "activity_fr.20160422104506",
         "small_vclock": 50,
         "w": "quorum",
         "young_vclock": 20
     }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
     "props": {
         "active": true,
         "allow_mult": false,
         "basic_quorum": false,
         "big_vclock": 50,
         "chash_keyfun": {
             "fun": "chash_std_keyfun",
             "mod": "riak_core_util"
         },
         "claimant": "[hidden email]",
         "dvv_enabled": false,
         "dw": "quorum",
         "last_write_wins": true,
         "linkfun": {
             "fun": "mapreduce_linkfun",
             "mod": "riak_kv_wm_link_walker"
         },
         "n_val": 3,
         "name": "tweet",
         "notfound_ok": true,
         "old_vclock": 86400,
         "postcommit": [],
         "pr": 0,
         "precommit": [],
         "pw": 0,
         "r": "quorum",
         "rw": "quorum",
         "search_index": "activity_fr.20160422104506",
         "small_vclock": 50,
         "w": "quorum",
         "young_vclock": 20
     }
}

I really don't know what to do. Can you help ?

Guillaume


On 02/05/2016 17:53, Luke Bakken wrote:

> Guillaume -
>
> Some colleagues had me carefully re-read those stats. You'll notice
> that those "put" stats are only for consistent or write_once
> operations, so they don't apply to you.
>
> Your read stats show objects well within Riak's recommended object size:
>
> node_get_fsm_objsize_100 : 10916
> node_get_fsm_objsize_95 : 7393
> node_get_fsm_objsize_99 : 8845
> node_get_fsm_objsize_mean : 4098
> node_get_fsm_objsize_median : 3891
>
> So that is not the issue.
>
> Are you using Bitcask? If so, please apply these sysctl settings:
>
> http://docs.basho.com/riak/kv/2.1.4/using/performance/#optional-i-o-settings
>
> If you are using the default "vm.dirty_*" settings Linux will appear
> to pause as it flushes disk buffers to the underlying device. The
> settings in the document change this so that flushes happen more often
> and asynchronously.
>
> --
> Luke Bakken
> Engineer
> [hidden email]
>
>
> On Mon, May 2, 2016 at 8:43 AM, Guillaume Boddaert
> <[hidden email]> wrote:
>> Here we go for a complete round of my hosts, all are objsize : 0
>>
>> Here is a sample answer (headers only, that are followed by the full set of
>> JSON content) from the RIAK5 host
>>
>> HTTP/1.1 200 OK
>> X-Riak-Vclock: a85hYGBgzGDKBVI8xTxKnGbpn7QYuPafyWBKZMxjZXjyYfYFviwA
>> Vary: Accept-Encoding
>> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
>> Link: </buckets/twitter>; rel="up"
>> Last-Modified: Mon, 02 May 2016 15:40:20 GMT
>> ETag: "2l2QODpewyBZQFqDnyEy3F"
>> Date: Mon, 02 May 2016 15:40:20 GMT
>> Content-Type: application/json
>> Content-Length: 10722
>>
>> Below the riak-admin status output.
>>
>>
>> admin@riak1:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>> admin@riak2:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>> admin@riak3:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>> admin@riak4:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>>
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>> admin@riak5:~$ sudo riak-admin status | grep -e 'objsize' | grep 'put'
>>
>> consistent_put_objsize_100 : 0
>> consistent_put_objsize_95 : 0
>> consistent_put_objsize_99 : 0
>> consistent_put_objsize_mean : 0
>> consistent_put_objsize_median : 0
>> write_once_put_objsize_100 : 0
>> write_once_put_objsize_95 : 0
>> write_once_put_objsize_99 : 0
>> write_once_put_objsize_mean : 0
>> write_once_put_objsize_median : 0
>>
>> On 02/05/2016 17:32, Luke Bakken wrote:
>>> Could you please check the objsize stats on every Riak node? If they
>>> are all zero then ... ????
>>> --
>>> Luke Bakken
>>> Engineer
>>> [hidden email]


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Luke Bakken
Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
<[hidden email]> wrote:

> Hi,
>
> Sorry for the delay, I've spent a lot of time trying to understand if the
> problem was elsewhere.
> I've simplified my infrastructure and got a simple layout that don't rely
> anymore on loadbalancer and also corrected some minor performance issue on
> my workers.
>
> At the moment, i have up to 32 workers that are calling riak for writes,
> each of them are set to :
> w=1
> dw=0
> timeout=1000
> using protobuf
> a timeouted attempt is rerun 180s later
>
> From my application server perspective, 23% of the calls are rejected by
> timeout (75446 tries, 57564 success, 17578 timeout).
>
> Here is a sample riak-admin stat for one of my 5 hosts:
>
> node_put_fsm_time_100 : 999331
> node_put_fsm_time_95 : 773682
> node_put_fsm_time_99 : 959444
> node_put_fsm_time_mean : 156242
> node_put_fsm_time_median : 20235
> vnode_put_fsm_time_100 : 5267527
> vnode_put_fsm_time_95 : 2437457
> vnode_put_fsm_time_99 : 4819538
> vnode_put_fsm_time_mean : 175567
> vnode_put_fsm_time_median : 6928
>
> I am using leveldb, so i can't tune bitcask backend as suggested.
>
> I've changed the vmdirty settings and enabled them:
> admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
> vm.dirty_background_bytes = 209715200
> vm.dirty_ratio = 40
> vm.dirty_bytes = 0
> vm.dirty_writeback_centisecs = 100
> vm.dirty_expire_centisecs = 200
>
> I've seen less idle time between writes, iostat is showing near constant
> writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
> better, but not that great.
>
> Here is the current configuration for my "activity_fr" bucket type and
> "tweet" bucket:
>
>
> admin@riak1:~$ http localhost:8098/types/activity_fr/props
> HTTP/1.1 200 OK
> Content-Encoding: gzip
> Content-Length: 314
> Content-Type: application/json
> Date: Tue, 03 May 2016 14:30:21 GMT
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Vary: Accept-Encoding
> {
>     "props": {
>         "active": true,
>         "allow_mult": false,
>         "basic_quorum": false,
>         "big_vclock": 50,
>         "chash_keyfun": {
>             "fun": "chash_std_keyfun",
>             "mod": "riak_core_util"
>         },
>         "claimant": "[hidden email]",
>         "dvv_enabled": false,
>         "dw": "quorum",
>         "last_write_wins": true,
>         "linkfun": {
>             "fun": "mapreduce_linkfun",
>             "mod": "riak_kv_wm_link_walker"
>         },
>         "n_val": 3,
>         "notfound_ok": true,
>         "old_vclock": 86400,
>         "postcommit": [],
>         "pr": 0,
>         "precommit": [],
>         "pw": 0,
>         "r": "quorum",
>         "rw": "quorum",
>         "search_index": "activity_fr.20160422104506",
>         "small_vclock": 50,
>         "w": "quorum",
>         "young_vclock": 20
>     }
> }
>
> admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
> HTTP/1.1 200 OK
> Content-Encoding: gzip
> Content-Length: 322
> Content-Type: application/json
> Date: Tue, 03 May 2016 14:30:02 GMT
> Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
> Vary: Accept-Encoding
>
> {
>     "props": {
>         "active": true,
>         "allow_mult": false,
>         "basic_quorum": false,
>         "big_vclock": 50,
>         "chash_keyfun": {
>             "fun": "chash_std_keyfun",
>             "mod": "riak_core_util"
>         },
>         "claimant": "[hidden email]",
>         "dvv_enabled": false,
>         "dw": "quorum",
>         "last_write_wins": true,
>         "linkfun": {
>             "fun": "mapreduce_linkfun",
>             "mod": "riak_kv_wm_link_walker"
>         },
>         "n_val": 3,
>         "name": "tweet",
>         "notfound_ok": true,
>         "old_vclock": 86400,
>         "postcommit": [],
>         "pr": 0,
>         "precommit": [],
>         "pw": 0,
>         "r": "quorum",
>         "rw": "quorum",
>         "search_index": "activity_fr.20160422104506",
>         "small_vclock": 50,
>         "w": "quorum",
>         "young_vclock": 20
>     }
> }
>
> I really don't know what to do. Can you help ?
>
> Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Matthew Von-Maszewski
Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <[hidden email]> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
<[hidden email]> wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": "[hidden email]",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": "[hidden email]",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Fred Dushin
Hi Guillaume,

From your bucket properties it looks like you are using search, and I assume that is search 2.0 (i.e., yokozuna), and not the legacy Riak Search.

It is true that in the current 2.0 and 2.1 trunks the indexing into Solr via Yokozuna is synchronous with the vnode -- very long times indexing can block vnodes while the indexing completes, which can increase latency for ordinary Riaks gets and puts.  You may be able to verify this by checking the mailbox lengths on your vnodes -- you may see messages enqueued while the Solr indexing completes. 

We are in the process of reviewing a PR [1] into the 2.0 trunk which adds batching and asynchronous indexing, decoupled from the riak vnodes responsible for initiating the work.  It's a significant chunk of work, and we are doing everything we can to ensure it is stable and performant.  Please note that even with batching and asynchronous writes into Solr, you will experience some increased latency and diminished throughput, as more work needs to be done on each node to index (and possibly store) data in Solr.

This work will eventually be forward merged into a post-2.0 branch.  Details will follow, once we get the work into 2.0.

I hope that helps.

-Fred


On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <[hidden email]> wrote:

Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <[hidden email]> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
<[hidden email]> wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": "[hidden email]",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": "[hidden email]",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Matthew Von-Maszewski
In reply to this post by Matthew Von-Maszewski
Guillaume,

A prebuilt eleveldb 2.0.18 for Debian 7 is found here:
   

There are good instructions for applying an eleveldb patch here:


Key points about the above web page:

- use the eleveldb patch file link in this email, NOT links on the web page

- the Debian directory listed on the web page will be slightly different than your Riak 2.1.4 installation:
    
    /usr/lib/riak/lib/eleveldb-<something_different>/priv/


Matthew


On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <[hidden email]> wrote:

Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <[hidden email]> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
<[hidden email]> wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": "[hidden email]",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": "[hidden email]",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
Thanks, I've installed the new library as stated in the documentation using 2.0.18 files.

I was unable to find the vnode LOG file from the documentation, as my vnodes looks like file, not directories. So I can't verify that I run the proper version of the library after my riak restart.

Anyway, it has unfortunately no effect:
http://www.awesomescreenshot.com/image/1219821/1b292613c051da86df5696034c114b14

I think i'll try to add a 6th node that don't rely on network disks and see what's going on.

G.


On 03/05/2016 22:47, Matthew Von-Maszewski wrote:
Guillaume,

A prebuilt eleveldb 2.0.18 for Debian 7 is found here:
   

There are good instructions for applying an eleveldb patch here:


Key points about the above web page:

- use the eleveldb patch file link in this email, NOT links on the web page

- the Debian directory listed on the web page will be slightly different than your Riak 2.1.4 installation:
    
    /usr/lib/riak/lib/eleveldb-<something_different>/priv/


Matthew


On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <[hidden email]> wrote:

Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <[hidden email]> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
[hidden email] wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": [hidden email],
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": [hidden email],
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Matthew Von-Maszewski
Guillaume,

Two points:

1.  You can send the “riak debug” from one server and I will verify that 2.0.18 is indicated in the LOG file.

2.  Your previous “riak debug” from server “riak1” indicated that only two CPU cores existed.  We performance test with eight, twelve, and twenty-four core servers, not two.  You have two heavy weight applications, Riak and Solr, competing for time on those two cores.  Actually, you have three applications due to leveldb’s background compaction operations.

One leveldb compaction is CPU intensive.  The compaction reads a block from the disk, computes a CRC32 check of the block, decompresses the block, merges the keys of this block with one or more blocks from other files, then compresses the new block, computes a new CRC32, and finally writes the block to disk.  And there can be multiple compactions running simultaneously.  All of your CPU time could be periodically lost to leveldb compactions.

There are some minor tunings we could do, like disabling compression in leveldb, that might help.  But I seriously doubt you are going to achieve your desired results with only two cores.  Adding a sixth server with two cores is not really going to help.

Matthew


On May 4, 2016, at 4:27 AM, Guillaume Boddaert <[hidden email]> wrote:

Thanks, I've installed the new library as stated in the documentation using 2.0.18 files.

I was unable to find the vnode LOG file from the documentation, as my vnodes looks like file, not directories. So I can't verify that I run the proper version of the library after my riak restart.

Anyway, it has unfortunately no effect:
http://www.awesomescreenshot.com/image/1219821/1b292613c051da86df5696034c114b14

I think i'll try to add a 6th node that don't rely on network disks and see what's going on.

G.


On 03/05/2016 22:47, Matthew Von-Maszewski wrote:
Guillaume,

A prebuilt eleveldb 2.0.18 for Debian 7 is found here:
   

There are good instructions for applying an eleveldb patch here:


Key points about the above web page:

- use the eleveldb patch file link in this email, NOT links on the web page

- the Debian directory listed on the web page will be slightly different than your Riak 2.1.4 installation:
    
    /usr/lib/riak/lib/eleveldb-<something_different>/priv/


Matthew


On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <[hidden email]> wrote:

Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <[hidden email]> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
[hidden email] wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": [hidden email],
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": [hidden email],
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Guillaume Boddaert
A little follow up for you guys since I went offline for quite some times.

As suggested, it was a Solr performance issue, we were able to prove that my old 5 hosts were able to handle the load without Solr/Yokozuna.
Fact was that I lacked CPU for my host, as well as RAM. Since SolR is pretty resource consuming, so I switched from :
- 5 x 16Gb x 2 CPU hosts
to
- 3 x 120Gb x 8 CPU hosts

And it now works like a charm,

Thanks for the help (especially to Damien)

Guillaume

On 04/05/2016 15:17, Matthew Von-Maszewski wrote:
Guillaume,

Two points:

1.  You can send the “riak debug” from one server and I will verify that 2.0.18 is indicated in the LOG file.

2.  Your previous “riak debug” from server “riak1” indicated that only two CPU cores existed.  We performance test with eight, twelve, and twenty-four core servers, not two.  You have two heavy weight applications, Riak and Solr, competing for time on those two cores.  Actually, you have three applications due to leveldb’s background compaction operations.

One leveldb compaction is CPU intensive.  The compaction reads a block from the disk, computes a CRC32 check of the block, decompresses the block, merges the keys of this block with one or more blocks from other files, then compresses the new block, computes a new CRC32, and finally writes the block to disk.  And there can be multiple compactions running simultaneously.  All of your CPU time could be periodically lost to leveldb compactions.

There are some minor tunings we could do, like disabling compression in leveldb, that might help.  But I seriously doubt you are going to achieve your desired results with only two cores.  Adding a sixth server with two cores is not really going to help.

Matthew


On May 4, 2016, at 4:27 AM, Guillaume Boddaert <[hidden email]> wrote:

Thanks, I've installed the new library as stated in the documentation using 2.0.18 files.

I was unable to find the vnode LOG file from the documentation, as my vnodes looks like file, not directories. So I can't verify that I run the proper version of the library after my riak restart.

Anyway, it has unfortunately no effect:
http://www.awesomescreenshot.com/image/1219821/1b292613c051da86df5696034c114b14

I think i'll try to add a 6th node that don't rely on network disks and see what's going on.

G.


On 03/05/2016 22:47, Matthew Von-Maszewski wrote:
Guillaume,

A prebuilt eleveldb 2.0.18 for Debian 7 is found here:
   

There are good instructions for applying an eleveldb patch here:


Key points about the above web page:

- use the eleveldb patch file link in this email, NOT links on the web page

- the Debian directory listed on the web page will be slightly different than your Riak 2.1.4 installation:
    
    /usr/lib/riak/lib/eleveldb-<something_different>/priv/


Matthew


On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <[hidden email]> wrote:

Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <[hidden email]> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
[hidden email]


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
[hidden email] wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": [hidden email],
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": [hidden email],
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com






_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Very slow acquisition time (99 percentile) while fast median times

Alexander Sicular-2
Awesome! Ya, Solr like resources. If you're on 3 nodes now, consider adjusting your n_val from default 3 to 2. With default ring_size of 64 and n_val of 3 and a cluster size less than 5 you are not guaranteed to have all copies of your data on distinct physical nodes. Some nodes will receive 2 copies of data. Just be aware of that. 

On Friday, May 27, 2016, Guillaume Boddaert <[hidden email]> wrote:
A little follow up for you guys since I went offline for quite some times.

As suggested, it was a Solr performance issue, we were able to prove that my old 5 hosts were able to handle the load without Solr/Yokozuna.
Fact was that I lacked CPU for my host, as well as RAM. Since SolR is pretty resource consuming, so I switched from :
- 5 x 16Gb x 2 CPU hosts
to
- 3 x 120Gb x 8 CPU hosts

And it now works like a charm,

Thanks for the help (especially to Damien)

Guillaume

On 04/05/2016 15:17, Matthew Von-Maszewski wrote:
Guillaume,

Two points:

1.  You can send the “riak debug” from one server and I will verify that 2.0.18 is indicated in the LOG file.

2.  Your previous “riak debug” from server “riak1” indicated that only two CPU cores existed.  We performance test with eight, twelve, and twenty-four core servers, not two.  You have two heavy weight applications, Riak and Solr, competing for time on those two cores.  Actually, you have three applications due to leveldb’s background compaction operations.

One leveldb compaction is CPU intensive.  The compaction reads a block from the disk, computes a CRC32 check of the block, decompresses the block, merges the keys of this block with one or more blocks from other files, then compresses the new block, computes a new CRC32, and finally writes the block to disk.  And there can be multiple compactions running simultaneously.  All of your CPU time could be periodically lost to leveldb compactions.

There are some minor tunings we could do, like disabling compression in leveldb, that might help.  But I seriously doubt you are going to achieve your desired results with only two cores.  Adding a sixth server with two cores is not really going to help.

Matthew


On May 4, 2016, at 4:27 AM, Guillaume Boddaert <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;guillaume@lighthouse-analytics.co&#39;);" target="_blank">guillaume@...> wrote:

Thanks, I've installed the new library as stated in the documentation using 2.0.18 files.

I was unable to find the vnode LOG file from the documentation, as my vnodes looks like file, not directories. So I can't verify that I run the proper version of the library after my riak restart.

Anyway, it has unfortunately no effect:
http://www.awesomescreenshot.com/image/1219821/1b292613c051da86df5696034c114b14

I think i'll try to add a 6th node that don't rely on network disks and see what's going on.

G.


On 03/05/2016 22:47, Matthew Von-Maszewski wrote:
Guillaume,

A prebuilt eleveldb 2.0.18 for Debian 7 is found here:
   

There are good instructions for applying an eleveldb patch here:


Key points about the above web page:

- use the eleveldb patch file link in this email, NOT links on the web page

- the Debian directory listed on the web page will be slightly different than your Riak 2.1.4 installation:
    
    /usr/lib/riak/lib/eleveldb-<something_different>/priv/


Matthew


On May 3, 2016, at 1:01 PM, Matthew Von-Maszewski <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;matthewv@basho.com&#39;);" target="_blank"><a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;matthewv@basho.com&#39;);" target="_blank">matthewv@...> wrote:

Guillaume,

I have reviewed the debug package for your riak1 server.  There are two potential areas of follow-up:

1.  You are running our most recent Riak 2.1.4 which has eleveldb 2.0.17.  We have seen a case where a recent feature in eleveldb 2.0.17 caused too much cache flushing, impacting leveldb’s performance.  A discussion is here:


2.  Yokozuna search was recently updated for some timeout problems.  Those updates are not yet in a public build.  One of our other engineers is likely to respond to you on that topic.


An eleveldb 2.0.18 is tagged and available via github if you want to build it yourself.  Otherwise, Basho may be releasing prebuilt patches of eleveldb 2.0.18 in the near future.  But no date is currently set.

Matthew

On May 3, 2016, at 10:50 AM, Luke Bakken <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;lbakken@basho.com&#39;);" target="_blank"><a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;lbakken@basho.com&#39;);" target="_blank">lbakken@...> wrote:

Guillaume -

You said earlier "My data are stored on an openstack volume that
support up to 3000IOPS". There is a likelihood that your write load is
exceeding the capacity of your virtual environment, especially if some
Riak nodes are sharing physical disk or server infrastructure.

Some suggestions:

* If you're not using Riak Search, set "search = off" in riak.conf

* Be sure to carefully read and apply all tunings:
http://docs.basho.com/riak/kv/2.1.4/using/performance/

* You may wish to increase the memory dedicated to leveldb:
http://docs.basho.com/riak/kv/2.1.4/configuring/backend/#leveldb

--
Luke Bakken
Engineer
<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;lbakken@basho.com&#39;);" target="_blank">lbakken@...


On Tue, May 3, 2016 at 7:33 AM, Guillaume Boddaert
<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;guillaume@lighthouse-analytics.co&#39;);" target="_blank"><guillaume@...> wrote:
Hi,

Sorry for the delay, I've spent a lot of time trying to understand if the
problem was elsewhere.
I've simplified my infrastructure and got a simple layout that don't rely
anymore on loadbalancer and also corrected some minor performance issue on
my workers.

At the moment, i have up to 32 workers that are calling riak for writes,
each of them are set to :
w=1
dw=0
timeout=1000
using protobuf
a timeouted attempt is rerun 180s later

From my application server perspective, 23% of the calls are rejected by
timeout (75446 tries, 57564 success, 17578 timeout).

Here is a sample riak-admin stat for one of my 5 hosts:

node_put_fsm_time_100 : 999331
node_put_fsm_time_95 : 773682
node_put_fsm_time_99 : 959444
node_put_fsm_time_mean : 156242
node_put_fsm_time_median : 20235
vnode_put_fsm_time_100 : 5267527
vnode_put_fsm_time_95 : 2437457
vnode_put_fsm_time_99 : 4819538
vnode_put_fsm_time_mean : 175567
vnode_put_fsm_time_median : 6928

I am using leveldb, so i can't tune bitcask backend as suggested.

I've changed the vmdirty settings and enabled them:
admin@riak1:~$ sudo sysctl -a | grep dirtyvm.dirty_background_ratio = 0
vm.dirty_background_bytes = 209715200
vm.dirty_ratio = 40
vm.dirty_bytes = 0
vm.dirty_writeback_centisecs = 100
vm.dirty_expire_centisecs = 200

I've seen less idle time between writes, iostat is showing near constant
writes between 20 and 500 kb/s, with some surges around 4000 kb/s. That's
better, but not that great.

Here is the current configuration for my "activity_fr" bucket type and
"tweet" bucket:


admin@riak1:~$ http localhost:8098/types/activity_fr/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 314
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:21 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding
{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak@riak2.lighthouse-analytics.co&#39;);" target="_blank"><a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak@riak2.lighthouse-analytics.co&#39;);" target="_blank">"riak@...",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

admin@riak1:~$ http localhost:8098/types/activity_fr/buckets/tweet/props
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Length: 322
Content-Type: application/json
Date: Tue, 03 May 2016 14:30:02 GMT
Server: MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)
Vary: Accept-Encoding

{
   "props": {
       "active": true,
       "allow_mult": false,
       "basic_quorum": false,
       "big_vclock": 50,
       "chash_keyfun": {
           "fun": "chash_std_keyfun",
           "mod": "riak_core_util"
       },
       "claimant": <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak@riak2.lighthouse-analytics.co&#39;);" target="_blank"><a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak@riak2.lighthouse-analytics.co&#39;);" target="_blank">"riak@...",
       "dvv_enabled": false,
       "dw": "quorum",
       "last_write_wins": true,
       "linkfun": {
           "fun": "mapreduce_linkfun",
           "mod": "riak_kv_wm_link_walker"
       },
       "n_val": 3,
       "name": "tweet",
       "notfound_ok": true,
       "old_vclock": 86400,
       "postcommit": [],
       "pr": 0,
       "precommit": [],
       "pw": 0,
       "r": "quorum",
       "rw": "quorum",
       "search_index": "activity_fr.20160422104506",
       "small_vclock": 50,
       "w": "quorum",
       "young_vclock": 20
   }
}

I really don't know what to do. Can you help ?

Guillaume

_______________________________________________
riak-users mailing list
<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;riak-users@lists.basho.com&#39;);" target="_blank">riak-users@...
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com







--


Alexander Sicular
Solutions Architect
Basho Technologies
9175130679
@siculars


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com