Riak overall performance falls down

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Riak overall performance falls down

mohammad sharifi
Hi
I have a clustered riak node, when I'm using basho bench to take a benchmark, performance falls down, (in spite of for example, 1K or 10K insert rate) after some time, overall insert rate is not smooth at all.
OS status after performance reduction: High memory usage, high user I/O wait
is there any one to explain why riak behaving like this?
thanks in advance


--
Mohammad Sharifi
Real Eyes Realize Real Lies

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Riak overall performance falls down

Travis Kirstine

Are you using leveldb?   Matthew (Basho) talk “Optimizing LevelDB for Performance and Scale” available in the link below provides a great overview

 

https://www.youtube.com/watch?v=vo88IdglU_8&t=1437s

 

The talk dates from 2013 and some things have changed but I found it very helpful

 

Have you gone through http://docs.basho.com/riak/kv/2.2.3/using/performance/

 

Besides that you’d need to provide more details on your setup

 

Regards

 

From: riak-users [mailto:[hidden email]] On Behalf Of mohammad sharifi
Sent: June-19-17 8:48 AM
To: [hidden email]
Subject: Riak overall performance falls down

 

Hi

I have a clustered riak node, when I'm using basho bench to take a benchmark, performance falls down, (in spite of for example, 1K or 10K insert rate) after some time, overall insert rate is not smooth at all.

OS status after performance reduction: High memory usage, high user I/O wait

is there any one to explain why riak behaving like this?

thanks in advance


 

--

Mohammad Sharifi
Real Eyes Realize Real Lies


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Riak overall performance falls down

mohammad sharifi
Hi
No I'm using bitcask

basho_ bench configurations are:
{mode, {rate, 300}}.

{duration, 4320}.

{report_interval, 1}.

{concurrent, 15}.

{driver, basho_bench_driver_riakc_pb}.

{key_generator, {int_to_bin_bigendian, {uniform_int, 1000000000}}}.

{value_generator, {fixed_bin, 1000}}.

{riakc_pb_ips, [{192,168,1,110},{192,168,1,111},{192,168,1,112}]}.

{riakc_pb_replies, 1}.

{riakc_pb_bucket, {<<"n_val_of_2">>,<<"bin-test">>}}.

{operations, [{put, 1}]}.

%% Use {auto_reconnect, false} to get "old" behavior (prior to April 2013).
%% See deps/riakc/src/riakc_pb_socket.erl for all valid socket options.
{pb_connect_options, [{auto_reconnect, true}]}.

%% Overrides for the PB client's default 60 second timeout, on a
%% per-type-of-operation basis.  All timeout units are specified in
%% milliseconds.  The pb_timeout_general config item provides a
%% default timeout if the read/write/listkeys/mapreduce timeout is not
%% specified.

{pb_timeout_general, 30000}.
{pb_timeout_read, 5000}.
{pb_timeout_write, 50000}.
{pb_timeout_listkeys, 50000}.
%% The general timeout will be used because this specific item is commented:
%% {pb_timeout_mapreduce, 50000}.
My OS and riak ocnfig:
cat /sys/kernel/mm/transparent_hugepage/enabled :
=================================================
always madvise [never]


cat /etc/fstab:
===============
/dev/sdb1       /mnt/storage    ext4    defaults,noatime,barrier=0,data=writeback      0      1


cat /sys/block/sda/queue/scheduler:
===================================
noop [deadline] cfq


cat /sys/block/sda/queue/nr_requests:
=====================================
128


cat /etc/sysctl.conf:
=====================
vm.swappiness=0
vm.dirty_background_ratio=0
vm.dirty_background_bytes=209715200
vm.dirty_ratio=0
vm.dirty_bytes=2000000000
vm.dirty_writeback_centisecs=100
vm.dirty_expire_centisecs=200

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

net.core.wmem_max=16777216
net.core.rmem_max=16777216
net.ipv4.tcp_mem=16777216 16777216 16777216
net.ipv4.tcp_rmem=10240 87380 16777216
net.ipv4.tcp_wmem=10240 87380 16777216
net.core.netdev_max_backlog=500000

net.core.rmem_default=16777216
net.core.wmem_default=16777216

net.ipv4.tcp_max_syn_backlog = 500000
net.core.somaxconn = 500000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_syncookies = 1

net.ipv4.ip_local_port_range=1024 655350
fs.file-max=10000000


cat /etc/riak/riak.conf:
========================
log.console = file
log.console.level = info
log.console.file = $(platform_log_dir)/console.log
log.error.file = $(platform_log_dir)/error.log
log.syslog = off
log.crash = on
log.crash.file = $(platform_log_dir)/crash.log
log.crash.maximum_message_size = 64KB
log.crash.size = 10MB
log.crash.rotation = $D0
log.crash.rotation.keep = 5
nodename = [hidden email]
distributed_cookie = riak
erlang.async_threads = 1000
erlang.max_ports = 262144
erlang.smp = auto
erlang.distribution_buffer_size = 2097151KB
erlang.async_threads.stack_size = 5MB
erlang.process_limit = 10000000
max_concurrent_requests = 3000000
background_manager = on
ring_size = 64
dtrace = off
platform_bin_dir = /usr/sbin
platform_data_dir = /var/lib/riak
platform_etc_dir = /etc/riak
platform_lib_dir = /usr/lib/riak/lib
platform_log_dir = /var/log/riak
listener.http.internal = 0.0.0.0:8098
listener.protobuf.internal = 192.168.1.111:8087
protobuf.backlog = 102400
anti_entropy = active
storage_backend = bitcask
object.format = 1
object.size.warning_threshold = 10MB
object.size.maximum = 50MB
object.siblings.warning_threshold = 25
object.siblings.maximum = 100
bitcask.data_root = /mnt/storage/bitcask
bitcask.io_mode = erlang
riak_control = off
riak_control.auth.mode = off
leveldb.maximum_memory.percent = 90
leveldb.compression = on
leveldb.compression.algorithm = lz4
search = off
search.solr.start_timeout = 30s
search.solr.port = 8093
search.solr.jmx_port = 8985
search.solr.jvm_options = -d64 -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
bitcask.max_file_size = 10GB
and my results are attached to mail.
thanks

On Mon, Jun 19, 2017 at 7:32 PM, Travis Kirstine <[hidden email]> wrote:

Are you using leveldb?   Matthew (Basho) talk “Optimizing LevelDB for Performance and Scale” available in the link below provides a great overview

 

https://www.youtube.com/watch?v=vo88IdglU_8&t=1437s

 

The talk dates from 2013 and some things have changed but I found it very helpful

 

Have you gone through http://docs.basho.com/riak/kv/2.2.3/using/performance/

 

Besides that you’d need to provide more details on your setup

 

Regards

 

From: riak-users [mailto:[hidden email]] On Behalf Of mohammad sharifi
Sent: June-19-17 8:48 AM
To: [hidden email]
Subject: Riak overall performance falls down

 

Hi

I have a clustered riak node, when I'm using basho bench to take a benchmark, performance falls down, (in spite of for example, 1K or 10K insert rate) after some time, overall insert rate is not smooth at all.

OS status after performance reduction: High memory usage, high user I/O wait

is there any one to explain why riak behaving like this?

thanks in advance


 

--

Mohammad Sharifi
Real Eyes Realize Real Lies




--
Mohammad Sharifi
Real Eyes Realize Real Lies

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

summary-12.png (125K) Download Attachment
summary-13.png (114K) Download Attachment
Loading...