Riak - low throughput

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak - low throughput

catchme
I have configured Riak using
3 nodes
n_val=3
dw=1
w=1
r=1
Using memory backend

The throughput is very low. Results after running the script for 10 mins
#samples  90% Line  Throughput
408679     17           683/sec

I am doing reads and writes to all the nodes.

How do I improve the throughput?

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

Mark Phillips-4
Hi There, 

A few questions to start:

* What type of hardware are these three nodes?
* Can you provide a gist of the script?
* What client library are you using?

Thanks, 

Mark 

On Wed, Feb 20, 2013 at 12:10 PM, catchme <[hidden email]> wrote:
I have configured Riak using
3 nodes
n_val=3
dw=1
w=1
r=1
Using memory backend

The throughput is very low. Results after running the script for 10 mins
#samples  90% Line  Throughput
408679     17           683/sec

I am doing reads and writes to all the nodes.

How do I improve the throughput?

Thanks.



--
View this message in context: http://riak-users.197444.n3.nabble.com/Riak-low-throughput-tp4026917.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

Dmitri Zagidulin
In reply to this post by catchme
When running throughput tests, I've usually found that the bottleneck is in the loading/testing script itself. That number (400-600 writes/sec) is usually the limit that a simple load script (consisting of a while loop that does PUTs to Riak) reaches.

Meaning, a single while loop issuing writes and waiting back for responses can't realistically write much faster than that.

The solution is - you have to bulk up your testing script, and introduce parallelism. Fire up several loading scripts in parallel. If you have multiple machines to write to the riak cluster, use those.  Make sure you're load-balancing your writes to all 3 nodes in the cluster, and not just hitting one. Use threads or worker pools to issue your writes, if that's available in your language.

General performance test bottleneck checklist (to get you past that 600 ops/sec):

1. Are you load balancing which nodes you're writing to, or always hitting the same node/IP? Either use an external balancer like HA proxy, and write to that, or use the round-robin balancer that's built into some of the Riak clients (like the Java client).

2. Is your test script single threaded? (Are you doing your writes linearly, in a while loop?) Spread the work to several different load scripts working in parallel. Spin up different threads to issue writes, if your language does multithreading well. If you're in Ruby, consider using worker pool managers like Celluloid, or something like Sidekiq.

There's other fine-tuning performance knobs having to do with backend settings, but those two above will get you the biggest gains.



On Wed, Feb 20, 2013 at 3:10 PM, catchme <[hidden email]> wrote:
I have configured Riak using
3 nodes
n_val=3
dw=1
w=1
r=1
Using memory backend

The throughput is very low. Results after running the script for 10 mins
#samples  90% Line  Throughput
408679     17           683/sec

I am doing reads and writes to all the nodes.

How do I improve the throughput?

Thanks.



--
View this message in context: http://riak-users.197444.n3.nabble.com/Riak-low-throughput-tp4026917.html
Sent from the Riak Users mailing list archive at Nabble.com.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

catchme
In reply to this post by Mark Phillips-4
* What type of hardware are these three nodes?
Using Redhat UNIX servers
4 vCPU  - Intel(R) Xeon(R) CPU;31GB memory
* Can you provide a gist of the script?
Using Jmeter to send the requests. Have 4 threads sending requests to these nodes continously
* What client library are you using?
Java protocol-buffer client
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

Alexander Sicular
Are these all virtual machines running on the same physical hardware? Cause that would be a problem...

-Alexander Sicular

@siculars

On Feb 21, 2013, at 8:40 AM, catchme wrote:

* What type of hardware are these three nodes?
Using Redhat UNIX servers
4 vCPU  - Intel(R) Xeon(R) CPU;31GB memory


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

catchme
Yes, they run on the same physical hardware.
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

Christian Dahlqvist
In reply to this post by catchme
Hi,

Does this mean that you are using a single connection to each node in your test? If this is the case I would recommend increasing the number of threads as Dmitri Zagidulin recommended in his response in order to get better throughput.

Also, what is the size of the objects you are writing?

Best regards,

Christian


On 21 Feb 2013, at 13:40, catchme <[hidden email]> wrote:

> * What type of hardware are these three nodes?
> Using Redhat UNIX servers
> 4 vCPU  - Intel(R) Xeon(R) CPU;31GB memory
> * Can you provide a gist of the script?
> Using Jmeter to send the requests. Have 4 threads sending requests to these
> nodes continously
> * What client library are you using?
> Java protocol-buffer client
>
>
>
>
> --
> View this message in context: http://riak-users.197444.n3.nabble.com/Riak-low-throughput-tp4026917p4026927.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak - low throughput

Richard Shaw
Hi,

If not done so already, It would be useful to run basho bench[0] on the cluster to see what the raw back end performance is


Regards

Richard


On Thu, Feb 21, 2013 at 2:37 PM, Christian Dahlqvist <[hidden email]> wrote:
Hi,

Does this mean that you are using a single connection to each node in your test? If this is the case I would recommend increasing the number of threads as Dmitri Zagidulin recommended in his response in order to get better throughput.

Also, what is the size of the objects you are writing?

Best regards,

Christian


On 21 Feb 2013, at 13:40, catchme <[hidden email]> wrote:

> * What type of hardware are these three nodes?
> Using Redhat UNIX servers
> 4 vCPU  - Intel(R) Xeon(R) CPU;31GB memory
> * Can you provide a gist of the script?
> Using Jmeter to send the requests. Have 4 threads sending requests to these
> nodes continously
> * What client library are you using?
> Java protocol-buffer client
>
>
>
>
> --
> View this message in context: http://riak-users.197444.n3.nabble.com/Riak-low-throughput-tp4026917p4026927.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com