Performance of write requests?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance of write requests?

Karsten Thygesen
Hi

I'm doing a small proof-of-concept and the goal is to store about 250.000.000 records in a Riak cluster. Today, we have the data in MySQL, but we strive for better performance and we might even expect up to 5 times as mush data during the next couple of years. The data is denormalized and "document" like so they are an easy match for NoSQL paradigm. 

For the small POC, I have built a 4 node cluster with 4 dedicated virtual servers running Opensolaris on top of VMWare but with quite fast storage below. In fron of the cluster I have a loadbalancer which will distribute reuests evenly among the nodes.

Each node is running riak-0.10 with almost deafult configuration. I have added "-smp enabled" to vm.args and each node is otherwise using default configuration (except for name of cause). This also implies N=2 and dest for storage backend.

I have written a small ruby script which uses riak-client from Ripple (latest version) as well as curd for http connections and it quite simple takes each record from the database and stores is in riak. Each record is around 500-1000 bytes large and entirely structured text/data. I store them as JSON objects.

The script can easily read more than 15.000 records/second, process them and print them to the screen, so I doubt the script is the bottleneck.

When I try to write them to the riak cluster via the loadbalancer, I can only write around 50-60 records/second and while writing, the beam process is only using  around 10% cpu and no major IO activity is going on.

I have tried to move the data directory to /tmp (memory filesystem) and with this setup, I can get around 90 write/sec (yes - only for testing - I can not live with memoryfilesystem in production with this dataset).

I have also noticed, that the performance I get is almost equivalent nomatter if I write through the loadbalancer or I just select a node and sends all my writes to that one. 

I have also tried a "multithreaded" approach where I simply run two of my datamover scripts in parallel, and that way, I can get around 110 writes/second.

With the current performance, it will take me more than a month to move my data from mysql to Riak, so I need a multitude of better performance.

Do you have any suggestions for how to get better performance? I was hoping for towards 1000 writes/second so feel free to speculate - perhaps I should just add quite a bunch of more servers?

Best regards,
Karsten

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance of write requests?

Mårten Gustafson
It'd be interesting to see what numbers you get if your script sets R & W = 1



 
/Mårten

On 10 maj 2010, at 17.36, Karsten Thygesen <[hidden email]> wrote:

Hi

I'm doing a small proof-of-concept and the goal is to store about 250.000.000 records in a Riak cluster. Today, we have the data in MySQL, but we strive for better performance and we might even expect up to 5 times as mush data during the next couple of years. The data is denormalized and "document" like so they are an easy match for NoSQL paradigm. 

For the small POC, I have built a 4 node cluster with 4 dedicated virtual servers running Opensolaris on top of VMWare but with quite fast storage below. In fron of the cluster I have a loadbalancer which will distribute reuests evenly among the nodes.

Each node is running riak-0.10 with almost deafult configuration. I have added "-smp enabled" to vm.args and each node is otherwise using default configuration (except for name of cause). This also implies N=2 and dest for storage backend.

I have written a small ruby script which uses riak-client from Ripple (latest version) as well as curd for http connections and it quite simple takes each record from the database and stores is in riak. Each record is around 500-1000 bytes large and entirely structured text/data. I store them as JSON objects.

The script can easily read more than 15.000 records/second, process them and print them to the screen, so I doubt the script is the bottleneck.

When I try to write them to the riak cluster via the loadbalancer, I can only write around 50-60 records/second and while writing, the beam process is only using  around 10% cpu and no major IO activity is going on.

I have tried to move the data directory to /tmp (memory filesystem) and with this setup, I can get around 90 write/sec (yes - only for testing - I can not live with memoryfilesystem in production with this dataset).

I have also noticed, that the performance I get is almost equivalent nomatter if I write through the loadbalancer or I just select a node and sends all my writes to that one. 

I have also tried a "multithreaded" approach where I simply run two of my datamover scripts in parallel, and that way, I can get around 110 writes/second.

With the current performance, it will take me more than a month to move my data from mysql to Riak, so I need a multitude of better performance.

Do you have any suggestions for how to get better performance? I was hoping for towards 1000 writes/second so feel free to speculate - perhaps I should just add quite a bunch of more servers?

Best regards,
Karsten
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance of write requests?

Stephan Maka
Mårten Gustafson wrote:
> It'd be interesting to see what numbers you get if your script sets R  
> & W = 1
> See: http://wiki.basho.com/display/RIAK/REST+API#RESTAPI-Storeaneworexistingobjectwithakey

That's only for the HTTP interface. Passing W=1 & DW=0 to
riak_client:put/3 has no effect. This is not yet a clustered
installation anyway.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance of write requests?

Ryan Tilder
In reply to this post by Karsten Thygesen
A couple of quick questions for you Karsten that should help us get an idea of what kind of issues you might be having.

How many physical hosts are you running the four OpenSolaris virtuals on?  If they're all running on the same host and you don't have a pretty substantial RAID array backing their local storage, you're just going to get I/O contention between the virtuals, slowing down writes.

There are some ZFS tuning parameters we've found that can improve write throughput.  Since you're using dets there's one in particular that will be helpful.  You can run this command as root on each OpenSolaris virtual:

zfs atime=off <pool>

The fact that you can essentially double your performance by running another client in parallel does make me wonder whether or not it might be a mild performance issue with your invocation of the ripple client.  Do you see a linear increase in write performance as you increase the number of parallel writers?

--Ryan

On Mon, May 10, 2010 at 8:36 AM, Karsten Thygesen <[hidden email]> wrote:
Hi

I'm doing a small proof-of-concept and the goal is to store about 250.000.000 records in a Riak cluster. Today, we have the data in MySQL, but we strive for better performance and we might even expect up to 5 times as mush data during the next couple of years. The data is denormalized and "document" like so they are an easy match for NoSQL paradigm. 

For the small POC, I have built a 4 node cluster with 4 dedicated virtual servers running Opensolaris on top of VMWare but with quite fast storage below. In fron of the cluster I have a loadbalancer which will distribute reuests evenly among the nodes.

Each node is running riak-0.10 with almost deafult configuration. I have added "-smp enabled" to vm.args and each node is otherwise using default configuration (except for name of cause). This also implies N=2 and dest for storage backend.

I have written a small ruby script which uses riak-client from Ripple (latest version) as well as curd for http connections and it quite simple takes each record from the database and stores is in riak. Each record is around 500-1000 bytes large and entirely structured text/data. I store them as JSON objects.

The script can easily read more than 15.000 records/second, process them and print them to the screen, so I doubt the script is the bottleneck.

When I try to write them to the riak cluster via the loadbalancer, I can only write around 50-60 records/second and while writing, the beam process is only using  around 10% cpu and no major IO activity is going on.

I have tried to move the data directory to /tmp (memory filesystem) and with this setup, I can get around 90 write/sec (yes - only for testing - I can not live with memoryfilesystem in production with this dataset).

I have also noticed, that the performance I get is almost equivalent nomatter if I write through the loadbalancer or I just select a node and sends all my writes to that one. 

I have also tried a "multithreaded" approach where I simply run two of my datamover scripts in parallel, and that way, I can get around 110 writes/second.

With the current performance, it will take me more than a month to move my data from mysql to Riak, so I need a multitude of better performance.

Do you have any suggestions for how to get better performance? I was hoping for towards 1000 writes/second so feel free to speculate - perhaps I should just add quite a bunch of more servers?

Best regards,
Karsten

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance of write requests?

Ryan Tilder
In reply to this post by Stephan Maka
By "not yet a clustered" installation do you mean that each of the 4 OpenSolaris nodes isn't communicating with the others?

--Ryan

On Mon, May 10, 2010 at 2:40 PM, Stephan Maka <[hidden email]> wrote:
Mårten Gustafson wrote:
> It'd be interesting to see what numbers you get if your script sets R
> & W = 1
> See: http://wiki.basho.com/display/RIAK/REST+API#RESTAPI-Storeaneworexistingobjectwithakey

That's only for the HTTP interface. Passing W=1 & DW=0 to
riak_client:put/3 has no effect. This is not yet a clustered
installation anyway.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
wde
Reply | Threaded
Open this post in threaded view
|

Re: Re: Performance of write requests?

wde
In reply to this post by Karsten Thygesen
do you have reference values by using an "in memory" storage backend for example, in order to clarify that the performance limit is related to the disk backend ?


wde


>A couple of quick questions for you Karsten that should help us get an idea
>of what kind of issues you might be having.
>
>How many physical hosts are you running the four OpenSolaris virtuals on?
> If they're all running on the same host and you don't have a pretty
>substantial RAID array backing their local storage, you're just going to get
>I/O contention between the virtuals, slowing down writes.
>
>There are some ZFS tuning parameters we've found that can improve write
>throughput.  Since you're using dets there's one in particular that will be
>helpful.  You can run this command as root on each OpenSolaris virtual:
>
>zfs atime=off <pool>
>
>The fact that you can essentially double your performance by running another
>client in parallel does make me wonder whether or not it might be a mild
>performance issue with your invocation of the ripple client.  Do you see a
>linear increase in write performance as you increase the number of parallel
>writers?
>
>--Ryan
>
>On Mon, May 10, 2010 at 8:36 AM, Karsten Thygesen <[hidden email]> wrote:
>
>> Hi
>>
>> I'm doing a small proof-of-concept and the goal is to store about
>> 250.000.000 records in a Riak cluster. Today, we have the data in MySQL, but
>> we strive for better performance and we might even expect up to 5 times as
>> mush data during the next couple of years. The data is denormalized and
>> "document" like so they are an easy match for NoSQL paradigm.
>>
>> For the small POC, I have built a 4 node cluster with 4 dedicated virtual
>> servers running Opensolaris on top of VMWare but with quite fast storage
>> below. In fron of the cluster I have a loadbalancer which will distribute
>> reuests evenly among the nodes.
>>
>> Each node is running riak-0.10 with almost deafult configuration. I have
>> added "-smp enabled" to vm.args and each node is otherwise using default
>> configuration (except for name of cause). This also implies N=2 and dest for
>> storage backend.
>>
>> I have written a small ruby script which uses riak-client from Ripple
>> (latest version) as well as curd for http connections and it quite simple
>> takes each record from the database and stores is in riak. Each record is
>> around 500-1000 bytes large and entirely structured text/data. I store them
>> as JSON objects.
>>
>> The script can easily read more than 15.000 records/second, process them
>> and print them to the screen, so I doubt the script is the bottleneck.
>>
>> When I try to write them to the riak cluster via the loadbalancer, I can
>> only write around 50-60 records/second and while writing, the beam process
>> is only using  around 10% cpu and no major IO activity is going on.
>>
>> I have tried to move the data directory to /tmp (memory filesystem) and
>> with this setup, I can get around 90 write/sec (yes - only for testing - I
>> can not live with memoryfilesystem in production with this dataset).
>>
>> I have also noticed, that the performance I get is almost equivalent
>> nomatter if I write through the loadbalancer or I just select a node and
>> sends all my writes to that one.
>>
>> I have also tried a "multithreaded" approach where I simply run two of my
>> datamover scripts in parallel, and that way, I can get around 110
>> writes/second.
>>
>> With the current performance, it will take me more than a month to move my
>> data from mysql to Riak, so I need a multitude of better performance.
>>
>> Do you have any suggestions for how to get better performance? I was hoping
>> for towards 1000 writes/second so feel free to speculate - perhaps I should
>> just add quite a bunch of more servers?
>>
>> Best regards,
>> *Karsten*
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>_______________________________________________
>riak-users mailing list
>[hidden email]
>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance of write requests?

Karsten Thygesen
In reply to this post by Ryan Tilder
Hi Ryan

I'm running the cluster on 4 physical servers with many cores each so CPU is not a limitation. Regarding the storage, we use Sun Openstorage which essential means, that we are writing to a flashdisk. Besides, I tried to use memorybased filesystem for storage with non significant changes in performance (50% better or so), so I doubt, that IO is my challenge. 

But I have noted the ZFS tuning recommendations - that will come in handy when we go in larger roll out.

But - David Dawson pointed me in the right direction - my simple converter script did a "fetch record, write record, repeat" lifecycle and the bare roundtrip time was a killer.

I have not extended my little script to become multithreaded (each write to the cluster happens in a dedicated thread) and that very simple enhancement now gave me more then 200 writes/sec, so I believe that the bottleneck is in my script and not the RIAK cluster right now.

I have run out of time for today, but I will continue my experiments tomorrow and finetune my script more (read: make it more parallel and perhaps distribute it to more hosts).

Best regards,
Karsten

On May 10, 2010, at 23:46 , Ryan Tilder wrote:

A couple of quick questions for you Karsten that should help us get an idea of what kind of issues you might be having.

How many physical hosts are you running the four OpenSolaris virtuals on?  If they're all running on the same host and you don't have a pretty substantial RAID array backing their local storage, you're just going to get I/O contention between the virtuals, slowing down writes.

There are some ZFS tuning parameters we've found that can improve write throughput.  Since you're using dets there's one in particular that will be helpful.  You can run this command as root on each OpenSolaris virtual:

zfs atime=off <pool>

The fact that you can essentially double your performance by running another client in parallel does make me wonder whether or not it might be a mild performance issue with your invocation of the ripple client.  Do you see a linear increase in write performance as you increase the number of parallel writers?

--Ryan

On Mon, May 10, 2010 at 8:36 AM, Karsten Thygesen <[hidden email]> wrote:
Hi

I'm doing a small proof-of-concept and the goal is to store about 250.000.000 records in a Riak cluster. Today, we have the data in MySQL, but we strive for better performance and we might even expect up to 5 times as mush data during the next couple of years. The data is denormalized and "document" like so they are an easy match for NoSQL paradigm. 

For the small POC, I have built a 4 node cluster with 4 dedicated virtual servers running Opensolaris on top of VMWare but with quite fast storage below. In fron of the cluster I have a loadbalancer which will distribute reuests evenly among the nodes.

Each node is running riak-0.10 with almost deafult configuration. I have added "-smp enabled" to vm.args and each node is otherwise using default configuration (except for name of cause). This also implies N=2 and dest for storage backend.

I have written a small ruby script which uses riak-client from Ripple (latest version) as well as curd for http connections and it quite simple takes each record from the database and stores is in riak. Each record is around 500-1000 bytes large and entirely structured text/data. I store them as JSON objects.

The script can easily read more than 15.000 records/second, process them and print them to the screen, so I doubt the script is the bottleneck.

When I try to write them to the riak cluster via the loadbalancer, I can only write around 50-60 records/second and while writing, the beam process is only using  around 10% cpu and no major IO activity is going on.

I have tried to move the data directory to /tmp (memory filesystem) and with this setup, I can get around 90 write/sec (yes - only for testing - I can not live with memoryfilesystem in production with this dataset).

I have also noticed, that the performance I get is almost equivalent nomatter if I write through the loadbalancer or I just select a node and sends all my writes to that one. 

I have also tried a "multithreaded" approach where I simply run two of my datamover scripts in parallel, and that way, I can get around 110 writes/second.

With the current performance, it will take me more than a month to move my data from mysql to Riak, so I need a multitude of better performance.

Do you have any suggestions for how to get better performance? I was hoping for towards 1000 writes/second so feel free to speculate - perhaps I should just add quite a bunch of more servers?

Best regards,
Karsten

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

smime.p7s (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Performance of write requests?

Karsten Thygesen
In reply to this post by Ryan Tilder
You are mixing Stephans and my setup - I do have a 4 noce Riak cluster running just fine - I believe Stephan have a single node?

Karsten

On May 11, 2010, at 0:05 , Ryan Tilder wrote:

By "not yet a clustered" installation do you mean that each of the 4 OpenSolaris nodes isn't communicating with the others?

--Ryan

On Mon, May 10, 2010 at 2:40 PM, Stephan Maka <[hidden email]> wrote:
Mårten Gustafson wrote:
> It'd be interesting to see what numbers you get if your script sets R
> & W = 1
> See: http://wiki.basho.com/display/RIAK/REST+API#RESTAPI-Storeaneworexistingobjectwithakey

That's only for the HTTP interface. Passing W=1 & DW=0 to
riak_client:put/3 has no effect. This is not yet a clustered
installation anyway.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

smime.p7s (2K) Download Attachment