Riak KV performance in sensor data storage application

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak KV performance in sensor data storage application

Yury Shevchuk
Greetings!

We have published a paper on the subject.  It may be of interest for
Riak developers and users who put Riak under heavy write load.

  English: http://psta.psiras.ru/read/psta2017_3_61-85.pdf
  Russian (native): http://psta.psiras.ru/read/psta2017_3_31-60.pdf

Best,


-- Yury

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak KV performance in sensor data storage application

Grigory Fateyev
Yury, thank you for sharing your investigations. It is very useful to read!

2017-10-03 8:32 GMT+03:00 Yury Shevchuk <[hidden email]>:
Greetings!

We have published a paper on the subject.  It may be of interest for
Riak developers and users who put Riak under heavy write load.

  English: http://psta.psiras.ru/read/psta2017_3_61-85.pdf
  Russian (native): http://psta.psiras.ru/read/psta2017_3_31-60.pdf

Best,


-- Yury

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak KV performance in sensor data storage application

DeadZen
This looks cool, and might give some indication that back pressure
should be aware of available file system space..
Any plans on LevelDB testing? A newer option Leveled might be cool to
compare with as well.

On Tue, Oct 3, 2017 at 6:58 AM, Grigory Fateyev <[hidden email]> wrote:

> Yury, thank you for sharing your investigations. It is very useful to read!
>
> 2017-10-03 8:32 GMT+03:00 Yury Shevchuk <[hidden email]>:
>>
>> Greetings!
>>
>> We have published a paper on the subject.  It may be of interest for
>> Riak developers and users who put Riak under heavy write load.
>>
>>   English: http://psta.psiras.ru/read/psta2017_3_61-85.pdf
>>   Russian (native): http://psta.psiras.ru/read/psta2017_3_31-60.pdf
>>
>> Best,
>>
>>
>> -- Yury
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak KV performance in sensor data storage application

Martin Sumner-2
I suspect with the test environment having only 4GB of RAM and a single slow disk per node - both Leveldb and Leveled are going to struggle in terms of the write throughput.  As n=1 in this test, some of the advantages of Leveled will be lost also (e.g. the n HEADs advantage).

I thought the paper was really interesting though.  It highlights an area of riak performance - merge effectiveness - that is not often discussed.  I've ignored merge effectiveness as a measure between leveldb and leveled, but it is an area where leveldb is currently notably better (and may well be better than Bitcask).  

As DeadZen mentioned having awareness of file system space is interesting.  The "fisherman's algorithm" approach mentioned in the paper is a neat, I've not thought about having a "Target Usage Level" - and I think this is potentially a better way of tuning a backend merge process.  I will try and put some effort in writing up the approach to tuning merge effectiveness that is implemented in leveled sometime soon for comparison.  


On 3 October 2017 at 17:32, DeadZen <[hidden email]> wrote:
This looks cool, and might give some indication that back pressure
should be aware of available file system space..
Any plans on LevelDB testing? A newer option Leveled might be cool to
compare with as well.

On Tue, Oct 3, 2017 at 6:58 AM, Grigory Fateyev <[hidden email]> wrote:
> Yury, thank you for sharing your investigations. It is very useful to read!
>
> 2017-10-03 8:32 GMT+03:00 Yury Shevchuk <[hidden email]>:
>>
>> Greetings!
>>
>> We have published a paper on the subject.  It may be of interest for
>> Riak developers and users who put Riak under heavy write load.
>>
>>   English: http://psta.psiras.ru/read/psta2017_3_61-85.pdf
>>   Russian (native): http://psta.psiras.ru/read/psta2017_3_31-60.pdf
>>
>> Best,
>>
>>
>> -- Yury
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


***** Email confidentiality notice *****
This message is private and confidential.  If you have received this message in error, please let us know and remove it from your system.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak KV performance in sensor data storage application

Yury Shevchuk
Greetings!

Thanks to everybody who cared to read the PSTA article.

DeadZen: we have no plans for LevelDB/LevelEd experiments for now.
We seem to have Bitcask doing what we need at the moment and switched
to application development.  Thanks for bringing LevelEd to our
attention, it looks promising for the future.

Martin: I have discovered the LevelEd treatise at
https://github.com/martinsumner/leveled/blob/master/docs/
and enjoy reading it.  Thank you!

On Tue, Oct 03, 2017 at 06:22:02PM +0100, Martin Sumner wrote:

> I suspect with the test environment having only 4GB of RAM and a single
> slow disk per node - both Leveldb and Leveled are going to struggle in
> terms of the write throughput.  As n=1 in this test, some of the advantages
> of Leveled will be lost also (e.g. the n HEADs advantage).
>
> I thought the paper was really interesting though.  It highlights an area
> of riak performance - merge effectiveness - that is not often discussed.
> I've ignored merge effectiveness as a measure between leveldb and leveled,
> but it is an area where leveldb is currently notably better (and may well
> be better than Bitcask).
>
> As DeadZen mentioned having awareness of file system space is interesting.
> The "fisherman's algorithm" approach mentioned in the paper is a neat, I've
> not thought about having a "Target Usage Level" - and I think this is
> potentially a better way of tuning a backend merge process.  I will try and
> put some effort in writing up the approach to tuning merge effectiveness
> that is implemented in leveled sometime soon for comparison.
>
>
> On 3 October 2017 at 17:32, DeadZen <[hidden email]> wrote:
>
> > This looks cool, and might give some indication that back pressure
> > should be aware of available file system space..
> > Any plans on LevelDB testing? A newer option Leveled might be cool to
> > compare with as well.
> >
> > On Tue, Oct 3, 2017 at 6:58 AM, Grigory Fateyev <[hidden email]> wrote:
> > > Yury, thank you for sharing your investigations. It is very useful to
> > read!
> > >
> > > 2017-10-03 8:32 GMT+03:00 Yury Shevchuk <[hidden email]>:
> > >>
> > >> Greetings!
> > >>
> > >> We have published a paper on the subject.  It may be of interest for
> > >> Riak developers and users who put Riak under heavy write load.
> > >>
> > >>   English: http://psta.psiras.ru/read/psta2017_3_61-85.pdf
> > >>   Russian (native): http://psta.psiras.ru/read/psta2017_3_31-60.pdf
> > >>
> > >> Best,
> > >>
> > >>
> > >> -- Yury

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com