Performance Tuning in OmniOS

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance Tuning in OmniOS

Hari John Kuriakose

I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS. The iops is very very low. There is no significant progress with tuning too.

Why I chose ZFS is that since LevelDB requires the node to be stopped before taking a backup, I needed a filesystem with snapshot ability. And the most favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris). Everything is fine, except the performance.

I did all the AWS tuning proposed by Basho but still Basho Bench gave twice iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am using riak-js client library, and its a 5 node Riak cluster with 8GB ram each.

Could not yet figure out what is really causing the congestion in OmniOS. Any pointers will be really helpful.

Thanks and regards,
Hari John Kuriakose.


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Hector Castro-2
Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector


On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose <[hidden email]> wrote:

>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Hari John Kuriakose
Hello,

I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.

I have made the following changes in zfs properties:

# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak

And I did the following with help from Basho AWS tuning docs:

projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak
bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"
bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000
ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608

Thanks again.


On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro <[hidden email]> wrote:
Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector


On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose <[hidden email]> wrote:
>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Jared Morrow
What type of RAID did you chose for your spool of 5 volumes?  If you chose the default of raidz, you will not be getting much of a performance boost over vanilla EBS, just a big integrity boost.  Also, unless you are using provisioned IOPS for EBS, you are starting from an extremely slow base-case, so adding ZFS on top might not help matters much.

If speed is the concern, as a test I'm willing to bet if you do another test run against the two instance storage disks on that m1.large, you will probably beat those 5 EBS volumes pretty easily.

-Jared


On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose <[hidden email]> wrote:
Hello,

I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.

I have made the following changes in zfs properties:

# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak

And I did the following with help from Basho AWS tuning docs:

projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak
bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"
bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000
ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608

Thanks again.


On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro <[hidden email]> wrote:
Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector


On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose <[hidden email]> wrote:
>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Tom Santero
this message from Hari John Kuriakose was accidentally auto-discarded. (sorry!)

From: Hari John Kuriakose <[hidden email]>
To: Jared Morrow <[hidden email]>
Cc: riak-users <[hidden email]>, Hector Castro <[hidden email]>
Date: Wed, 22 Jan 2014 00:35:37 +0530
Subject: Re: Performance Tuning in OmniOS

I am using the default raid itself.

Well, if this is the case, I will run the tests again with a different setup as you said, and get back as soon as possible. I would just like to believe that OmniOS is not too hopeless.

Thank you.




On Tue, Jan 21, 2014 at 12:47 PM, Jared Morrow <[hidden email]> wrote:
What type of RAID did you chose for your spool of 5 volumes?  If you chose the default of raidz, you will not be getting much of a performance boost over vanilla EBS, just a big integrity boost.  Also, unless you are using provisioned IOPS for EBS, you are starting from an extremely slow base-case, so adding ZFS on top might not help matters much.

If speed is the concern, as a test I'm willing to bet if you do another test run against the two instance storage disks on that m1.large, you will probably beat those 5 EBS volumes pretty easily.

-Jared


On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose <[hidden email]> wrote:
Hello,

I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.

I have made the following changes in zfs properties:

# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak

And I did the following with help from Basho AWS tuning docs:

projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak
bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"
bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000
ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608

Thanks again.


On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro <[hidden email]> wrote:
Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector


On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose <[hidden email]> wrote:
>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Jared Morrow
In reply to this post by Jared Morrow
Oh I think OmniOS is far from hopeless.  The problem you are having is the same problem you'd have if you were on ubuntu and you made a LVM raid on vanilla EBS.  EBS is the problem when it comes to predictable write / read speed.  People still use it, but not without careful thought and consideration.  You can try using provisioned IOPS for EBS, which the m1.large supports, or ask in risk-users what other AWS users have setup.  I know we have a lot of customers and OSS users running on AWS, so they are far more knowledgeable about real-world performance than I am.

Good luck,
Jared




On Tue, Jan 21, 2014 at 12:05 PM, Hari John Kuriakose <[hidden email]> wrote:

I am using the default raid itself.

Well, if this is the case, I will run the tests again with a different setup as you said, and get back as soon as possible. I would just like to believe that OmniOS is not too hopeless.

Thank you.

On Jan 21, 2014 11:17 PM, "Jared Morrow" <[hidden email]> wrote:
What type of RAID did you chose for your spool of 5 volumes?  If you chose the default of raidz, you will not be getting much of a performance boost over vanilla EBS, just a big integrity boost.  Also, unless you are using provisioned IOPS for EBS, you are starting from an extremely slow base-case, so adding ZFS on top might not help matters much.

If speed is the concern, as a test I'm willing to bet if you do another test run against the two instance storage disks on that m1.large, you will probably beat those 5 EBS volumes pretty easily.

-Jared


On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose <[hidden email]> wrote:
Hello,

I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.

I have made the following changes in zfs properties:

# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak

And I did the following with help from Basho AWS tuning docs:

projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak
bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"
bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000
ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608

Thanks again.


On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro <[hidden email]> wrote:
Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector


On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose <[hidden email]> wrote:
>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Jason Campbell-2
First off, make sure you are comparing apples to apples.  I am assuming the "default" RAID is RAIDZ on ZFS, so make sure the LVM raid is using RAID5 for comparable performance.

Generally, I wouldn't be using RAIDZ on AWS at all.  The primary issue with RAID5/RAIDZ is that the IO speeds are not only limited to that of a single EBS volume, but that of the slowest EBS volume.  That speed can vary with which host you get assigned to, time of day, contention from other customers, and seemingly the phase of the moon.  I don't think anyone would be surprised if you ran the same test again and got completely different results.  Provisioned IOPS disks will help even out extremely slow disks, but you can still get quite a bit of variance.

I would suggest moving to mirrored disks (RAID1) in both ZFS and Ubuntu.  I'm not sure about LVM, but ZFS will use the mirror to even out reads (writes are harder) which should fix some of the high latency, even on normal EBS volumes.  I would suggest 4 disks in a RAID 10 configuration (striping 2 mirrored pairs).  Even better would be a 4-way mirror in ZFS if price isn't much of a concern.  This will limit you to the capacity of a single EBS volume, but reads will use the fastest disk of the 4 disks, instead of the slowest.  It also has a nice side effect of being extremely fault tolerant.

The other thing to keep in mind is that ZFS is extremely RAM hungry and read performance drops considerably when it is RAM starved, so I would ensure that Riak doesn't use the last 1 - 1.5 GB of RAM so ZFS can use it for caches.

So in summary:
 - Test, test and test again on different instances, this will help even out EBS issues.
 - Use mirrors, not RAID5/RAIDZ
 - Use dedicated IOPS (at least for testing purposes)
 - Ensure ZFS has some RAM to play with

Hope this helps,
Jason Campbell

----- Original Message -----
From: "Jared Morrow" <[hidden email]>
To: [hidden email]
Cc: "riak-users" <[hidden email]>
Sent: Wednesday, 22 January, 2014 7:51:19 AM
Subject: Re: Performance Tuning in OmniOS



Oh I think OmniOS is far from hopeless. The problem you are having is the same problem you'd have if you were on ubuntu and you made a LVM raid on vanilla EBS. EBS is the problem when it comes to predictable write / read speed. People still use it, but not without careful thought and consideration. You can try using provisioned IOPS for EBS, which the m1.large supports, or ask in risk-users what other AWS users have setup. I know we have a lot of customers and OSS users running on AWS, so they are far more knowledgeable about real-world performance than I am.


Good luck,
Jared








On Tue, Jan 21, 2014 at 12:05 PM, Hari John Kuriakose < [hidden email] > wrote:




I am using the default raid itself.

Well, if this is the case, I will run the tests again with a different setup as you said, and get back as soon as possible. I would just like to believe that OmniOS is not too hopeless.

Thank you.


On Jan 21, 2014 11:17 PM, "Jared Morrow" < [hidden email] > wrote:



What type of RAID did you chose for your spool of 5 volumes? If you chose the default of raidz, you will not be getting much of a performance boost over vanilla EBS, just a big integrity boost. Also, unless you are using provisioned IOPS for EBS, you are starting from an extremely slow base-case, so adding ZFS on top might not help matters much.


If speed is the concern, as a test I'm willing to bet if you do another test run against the two instance storage disks on that m1.large, you will probably beat those 5 EBS volumes pretty easily.


-Jared



On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose < [hidden email] > wrote:



Hello,


I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.


I have made the following changes in zfs properties:



# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak


And I did the following with help from Basho AWS tuning docs:


projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak

bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"


bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000


ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608


Thanks again.





On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro < [hidden email] > wrote:


Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector




On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose < [hidden email] > wrote:

>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Hari John Kuriakose
Thanks again for the quick reply.

As you said, I am wondering about apples and oranges primarily. The same EBS volume backed setup on Linux beat the OmniOS setup my more than 2x. That is more weird to me than normal. This I tested with simple RAID0 (40Gb * 5 EBS devices) on both the operating systems.

Also, I had followed the LevelDB tuning docs and set 50% of 8GB ram for LevelDB, and rest was supposed to be for os. But the ZFS ARC cache seems to have a mind of its own. This is unlike how LInux efficiently manages a disk cache automatically. Again quoting your words, ZFS do needs ram, but I find a lack of proper documentation on the ARC cache atleast so that I could tune it specifically for AWS.

Finally, I would want my RAID setup to reduce the latency rather than to be fault tolerant, since I hope Riak will be available. Also if RAIDZ is going to have trade-offs of its own, I would not want that to add up to the unreliability of standard EBS volumes. As summarized, I would repeat my tests focusing on maybe PIOPS because price is also a concern.

Ultimately, I just wish to avoid any common pitfalls or dead-ends.

Regards,
Hari John Kuriakose.


On Wed, Jan 22, 2014 at 3:26 AM, Jason Campbell <[hidden email]> wrote:
First off, make sure you are comparing apples to apples.  I am assuming the "default" RAID is RAIDZ on ZFS, so make sure the LVM raid is using RAID5 for comparable performance.

Generally, I wouldn't be using RAIDZ on AWS at all.  The primary issue with RAID5/RAIDZ is that the IO speeds are not only limited to that of a single EBS volume, but that of the slowest EBS volume.  That speed can vary with which host you get assigned to, time of day, contention from other customers, and seemingly the phase of the moon.  I don't think anyone would be surprised if you ran the same test again and got completely different results.  Provisioned IOPS disks will help even out extremely slow disks, but you can still get quite a bit of variance.

I would suggest moving to mirrored disks (RAID1) in both ZFS and Ubuntu.  I'm not sure about LVM, but ZFS will use the mirror to even out reads (writes are harder) which should fix some of the high latency, even on normal EBS volumes.  I would suggest 4 disks in a RAID 10 configuration (striping 2 mirrored pairs).  Even better would be a 4-way mirror in ZFS if price isn't much of a concern.  This will limit you to the capacity of a single EBS volume, but reads will use the fastest disk of the 4 disks, instead of the slowest.  It also has a nice side effect of being extremely fault tolerant.

The other thing to keep in mind is that ZFS is extremely RAM hungry and read performance drops considerably when it is RAM starved, so I would ensure that Riak doesn't use the last 1 - 1.5 GB of RAM so ZFS can use it for caches.

So in summary:
 - Test, test and test again on different instances, this will help even out EBS issues.
 - Use mirrors, not RAID5/RAIDZ
 - Use dedicated IOPS (at least for testing purposes)
 - Ensure ZFS has some RAM to play with

Hope this helps,
Jason Campbell

----- Original Message -----
From: "Jared Morrow" <[hidden email]>
To: [hidden email]
Cc: "riak-users" <[hidden email]>
Sent: Wednesday, 22 January, 2014 7:51:19 AM
Subject: Re: Performance Tuning in OmniOS



Oh I think OmniOS is far from hopeless. The problem you are having is the same problem you'd have if you were on ubuntu and you made a LVM raid on vanilla EBS. EBS is the problem when it comes to predictable write / read speed. People still use it, but not without careful thought and consideration. You can try using provisioned IOPS for EBS, which the m1.large supports, or ask in risk-users what other AWS users have setup. I know we have a lot of customers and OSS users running on AWS, so they are far more knowledgeable about real-world performance than I am.


Good luck,
Jared








On Tue, Jan 21, 2014 at 12:05 PM, Hari John Kuriakose < [hidden email] > wrote:




I am using the default raid itself.

Well, if this is the case, I will run the tests again with a different setup as you said, and get back as soon as possible. I would just like to believe that OmniOS is not too hopeless.

Thank you.


On Jan 21, 2014 11:17 PM, "Jared Morrow" < [hidden email] > wrote:



What type of RAID did you chose for your spool of 5 volumes? If you chose the default of raidz, you will not be getting much of a performance boost over vanilla EBS, just a big integrity boost. Also, unless you are using provisioned IOPS for EBS, you are starting from an extremely slow base-case, so adding ZFS on top might not help matters much.


If speed is the concern, as a test I'm willing to bet if you do another test run against the two instance storage disks on that m1.large, you will probably beat those 5 EBS volumes pretty easily.


-Jared



On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose < [hidden email] > wrote:



Hello,


I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.


I have made the following changes in zfs properties:



# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak


And I did the following with help from Basho AWS tuning docs:


projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak

bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"


bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000


ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608


Thanks again.





On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro < [hidden email] > wrote:


Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector




On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose < [hidden email] > wrote:
>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Performance Tuning in OmniOS

Hari John Kuriakose
Thought I would clarify my suspicion regarding the performance difference between Linux and OmniOS.

Turns out during my testing with OmniOS, I did not launch some nodes as Amazon "EBS Optimized". This was the primary reason for the comparitively very poor IOPS obtained in OmniOS setup.

I would also like to add that even though I had a nice run with Riak on OmniOS with ZFS, I finally settled with Ubuntu itself. Linux has always been familiar territory.

Ofcourse other things like snapshots (I used LevelDB as backend), RAID config, etc. were taken care of in other ways but atleast there was nothing unknown or weird anymore.

Hope this would clear of any wrong impression I may have created, and may help someone in some way.

Thanks once again to all of you for your help.

Regards,
Hari John Kuriakose.


On Wed, Jan 22, 2014 at 4:15 PM, Hari John Kuriakose <[hidden email]> wrote:
Thanks again for the quick reply.

As you said, I am wondering about apples and oranges primarily. The same EBS volume backed setup on Linux beat the OmniOS setup my more than 2x. That is more weird to me than normal. This I tested with simple RAID0 (40Gb * 5 EBS devices) on both the operating systems.

Also, I had followed the LevelDB tuning docs and set 50% of 8GB ram for LevelDB, and rest was supposed to be for os. But the ZFS ARC cache seems to have a mind of its own. This is unlike how LInux efficiently manages a disk cache automatically. Again quoting your words, ZFS do needs ram, but I find a lack of proper documentation on the ARC cache atleast so that I could tune it specifically for AWS.

Finally, I would want my RAID setup to reduce the latency rather than to be fault tolerant, since I hope Riak will be available. Also if RAIDZ is going to have trade-offs of its own, I would not want that to add up to the unreliability of standard EBS volumes. As summarized, I would repeat my tests focusing on maybe PIOPS because price is also a concern.

Ultimately, I just wish to avoid any common pitfalls or dead-ends.

Regards,
Hari John Kuriakose.


On Wed, Jan 22, 2014 at 3:26 AM, Jason Campbell <[hidden email]> wrote:
First off, make sure you are comparing apples to apples.  I am assuming the "default" RAID is RAIDZ on ZFS, so make sure the LVM raid is using RAID5 for comparable performance.

Generally, I wouldn't be using RAIDZ on AWS at all.  The primary issue with RAID5/RAIDZ is that the IO speeds are not only limited to that of a single EBS volume, but that of the slowest EBS volume.  That speed can vary with which host you get assigned to, time of day, contention from other customers, and seemingly the phase of the moon.  I don't think anyone would be surprised if you ran the same test again and got completely different results.  Provisioned IOPS disks will help even out extremely slow disks, but you can still get quite a bit of variance.

I would suggest moving to mirrored disks (RAID1) in both ZFS and Ubuntu.  I'm not sure about LVM, but ZFS will use the mirror to even out reads (writes are harder) which should fix some of the high latency, even on normal EBS volumes.  I would suggest 4 disks in a RAID 10 configuration (striping 2 mirrored pairs).  Even better would be a 4-way mirror in ZFS if price isn't much of a concern.  This will limit you to the capacity of a single EBS volume, but reads will use the fastest disk of the 4 disks, instead of the slowest.  It also has a nice side effect of being extremely fault tolerant.

The other thing to keep in mind is that ZFS is extremely RAM hungry and read performance drops considerably when it is RAM starved, so I would ensure that Riak doesn't use the last 1 - 1.5 GB of RAM so ZFS can use it for caches.

So in summary:
 - Test, test and test again on different instances, this will help even out EBS issues.
 - Use mirrors, not RAID5/RAIDZ
 - Use dedicated IOPS (at least for testing purposes)
 - Ensure ZFS has some RAM to play with

Hope this helps,
Jason Campbell

----- Original Message -----
From: "Jared Morrow" <[hidden email]>
To: [hidden email]
Cc: "riak-users" <[hidden email]>
Sent: Wednesday, 22 January, 2014 7:51:19 AM
Subject: Re: Performance Tuning in OmniOS



Oh I think OmniOS is far from hopeless. The problem you are having is the same problem you'd have if you were on ubuntu and you made a LVM raid on vanilla EBS. EBS is the problem when it comes to predictable write / read speed. People still use it, but not without careful thought and consideration. You can try using provisioned IOPS for EBS, which the m1.large supports, or ask in risk-users what other AWS users have setup. I know we have a lot of customers and OSS users running on AWS, so they are far more knowledgeable about real-world performance than I am.


Good luck,
Jared








On Tue, Jan 21, 2014 at 12:05 PM, Hari John Kuriakose < [hidden email] > wrote:




I am using the default raid itself.

Well, if this is the case, I will run the tests again with a different setup as you said, and get back as soon as possible. I would just like to believe that OmniOS is not too hopeless.

Thank you.


On Jan 21, 2014 11:17 PM, "Jared Morrow" < [hidden email] > wrote:



What type of RAID did you chose for your spool of 5 volumes? If you chose the default of raidz, you will not be getting much of a performance boost over vanilla EBS, just a big integrity boost. Also, unless you are using provisioned IOPS for EBS, you are starting from an extremely slow base-case, so adding ZFS on top might not help matters much.


If speed is the concern, as a test I'm willing to bet if you do another test run against the two instance storage disks on that m1.large, you will probably beat those 5 EBS volumes pretty easily.


-Jared



On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose < [hidden email] > wrote:



Hello,


I am using standard EBS devices, with a zpool in an instance comprising of five 40GB volumes.
Each of the Riak instance is of m1.large type.


I have made the following changes in zfs properties:



# My reason: the default sst block size for leveldb is 4k.
zfs set recordsize=4k tank/riak
# My reason: by default, leveldb verifies checksums automatically.
zfs set checksum=off tank/riak
zfs set atime=off tank/riak
zfs set snapdir=visible tank/riak


And I did the following with help from Basho AWS tuning docs:


projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" user.riak

bash -c "echo 'set rlim_fd_max=65536' >> /etc/system"


bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system"
ndd -set /dev/tcp tcp_conn_req_max_q0 40000


ndd -set /dev/tcp tcp_conn_req_max_q 4000
ndd -set /dev/tcp tcp_tstamp_always 0
ndd -set /dev/tcp tcp_sack_permitted 2
ndd -set /dev/tcp tcp_wscale_always 1
ndd -set /dev/tcp tcp_time_wait_interval 60000
ndd -set /dev/tcp tcp_keepalive_interval 120000
ndd -set /dev/tcp tcp_xmit_hiwat 2097152
ndd -set /dev/tcp tcp_recv_hiwat 2097152
ndd -set /dev/tcp tcp_max_buf 8388608


Thanks again.





On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro < [hidden email] > wrote:


Hello,

Can you please clarify what type of disk you are using within AWS?
EBS, EBS with PIOPS, instance storage? In addition, maybe some details
on volume sizes and instance types.

These details may help someone attempting to answer your question.

--
Hector




On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose < [hidden email] > wrote:
>
> I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon AWS.
> The iops is very very low. There is no significant progress with tuning too.
>
> Why I chose ZFS is that since LevelDB requires the node to be stopped before
> taking a backup, I needed a filesystem with snapshot ability. And the most
> favourable Amazon community AMI seemed to be using OmniOS (fork of Solaris).
> Everything is fine, except the performance.
>
> I did all the AWS tuning proposed by Basho but still Basho Bench gave twice
> iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am
> using riak-js client library, and its a 5 node Riak cluster with 8GB ram
> each.
>
> Could not yet figure out what is really causing the congestion in OmniOS.
> Any pointers will be really helpful.
>
> Thanks and regards,
> Hari John Kuriakose.
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com