LevelDB driver

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

LevelDB driver

Dave Smith
Hi all,

Yesterday, we merged in support for the LevelDB
(http://code.google.com/p/leveldb/) as a backend driver for Riak.

Initial benchmarking of LevelDB suggests that it is competitive to
InnoDB (aka innostore) for most use cases. The recovery model is much
better, as LevelDB uses an append-only storage system, and the license
is such that we can include it in the default Riak packages (i.e.
much, much easier to use).

To get started, grab the latest Riak from Github and use the
"riak_kv_eleveldb_backend" in app.config as your backend driver. Data
will be stored in data/leveldb.

We anticipate LevelDB being a good complement to Bitcask. For anything
approaching random I/O, where it's acceptable for the keyspace to be
in RAM, bitcask will be very (very) hard to beat. However, if you've
got a large keyspace that doesn't have purely random I/O, LevelDB will
useful and meet or exceed the performance characteristics seen with
Inno. You can find a more complete analysis on our blog:

http://blog.basho.com/2011/07/01/Leveling-the-Field/

As with some of the other major changes landing this summer, we'd
encourage you NOT to use this in production just yet.

Thanks,

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Will Moss
Hey Dave,

This is very cool--glad you guys decided to bundle this in. The linked post and the Google Code page both suggest that it will have a much more efficient on-disk representation than InnoDB, do you have an specific numbers on overhead per key?

Thanks,
Will
 

On Fri, Jul 1, 2011 at 9:03 AM, David Smith <[hidden email]> wrote:
Hi all,

Yesterday, we merged in support for the LevelDB
(http://code.google.com/p/leveldb/) as a backend driver for Riak.

Initial benchmarking of LevelDB suggests that it is competitive to
InnoDB (aka innostore) for most use cases. The recovery model is much
better, as LevelDB uses an append-only storage system, and the license
is such that we can include it in the default Riak packages (i.e.
much, much easier to use).

To get started, grab the latest Riak from Github and use the
"riak_kv_eleveldb_backend" in app.config as your backend driver. Data
will be stored in data/leveldb.

We anticipate LevelDB being a good complement to Bitcask. For anything
approaching random I/O, where it's acceptable for the keyspace to be
in RAM, bitcask will be very (very) hard to beat. However, if you've
got a large keyspace that doesn't have purely random I/O, LevelDB will
useful and meet or exceed the performance characteristics seen with
Inno. You can find a more complete analysis on our blog:

http://blog.basho.com/2011/07/01/Leveling-the-Field/

As with some of the other major changes landing this summer, we'd
encourage you NOT to use this in production just yet.

Thanks,

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Dave Smith
On Fri, Jul 1, 2011 at 12:53 PM, Will Moss <[hidden email]> wrote:
> This is very cool--glad you guys decided to bundle this in. The linked post
> and the Google Code page both suggest that it will have a much more
> efficient on-disk representation than InnoDB, do you have an specific
> numbers on overhead per key?

I don't think we've officially measured it; in my own tests, it's
quite small. A 10GB dataset was within a few MB of what I expected.
Certainly FAR more efficient than Inno (where I've seen 2-3x).

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Jonathan Langevin
I've seen users show concern of Bitcask's space usage overhead. How does that compare against LevelDB?
Would LevelDB be a good solution for log data?

If using a Level backend, what advantages do we lose of Bitcask? ls replication & availability an issue at all?


Jonathan Langevin
Systems Administrator

Loom Inc.
Wilmington, NC: (910) 241-0433 - [hidden email] - www.loomlearning.com - Skype: intel352



On Fri, Jul 1, 2011 at 2:58 PM, David Smith <[hidden email]> wrote:
On Fri, Jul 1, 2011 at 12:53 PM, Will Moss <[hidden email]> wrote:
> This is very cool--glad you guys decided to bundle this in. The linked post
> and the Google Code page both suggest that it will have a much more
> efficient on-disk representation than InnoDB, do you have an specific
> numbers on overhead per key?

I don't think we've officially measured it; in my own tests, it's
quite small. A 10GB dataset was within a few MB of what I expected.
Certainly FAR more efficient than Inno (where I've seen 2-3x).

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Justin Sheehy
Hi, Jonathan.

On Mon, Jul 4, 2011 at 9:42 AM, Jonathan Langevin
<[hidden email]> wrote:

> I've seen users show concern of Bitcask's space usage overhead. How does that
> compare against LevelDB?

Bitcask doesn't have much in the way of disk space "overhead" unless
you mean that the space used by deleted or overwritten values is not
reclaimed until after a merge. In that way LevelDB is similar since
space used by deleted and overwritten items is reclaimed as they are
moved into older "levels" of the DB. The behavior here is not
identical, but similar in concept.

By way of comparison, InnoDB imposes about a 2x space overhead cost on
many common datasets but the overhead is usually fairly static.

> If using a Level backend, what advantages do we lose of Bitcask? ls replication &
> availability an issue at all?

The functionality provided by Riak above the storage engines (such as
replication and system-wide availability) are generally not impacted
by your choice of storage engine.

There are two main things you would lose today:

1 - latency
2 - stability

The first of these is fundamental: for many usage patterns Bitcask
will have a latency advantage over LevelDB due to being able to
guarantee that it will never perform more than a single disk seek per
operation.

The second is just about the relative immaturity of LevelDB: we have
not yet seen LevelDB in production environments for an extended amount
of time as we have with Bitcask. Anyone using it now as a Bitcask
replacement should realize that they are on the leading edge and
taking the usual risks that come with adopting new software. That
said, we expect LevelDB to do well over time as one of the alternative
Riak storage engines.

The main reason to use LevelDB under Riak would be if your number of
keys is huge and thus the RAM consumption of Bitcask would make it
unsuitable. That is, we expect people to use LevelDB in the same
situations that they might previously have chosen Innostore as their
storage engine.

-Justin

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Jonathan Langevin
Thanks Justin for the helpful response :-)

Can you define what you would consider "huge" regarding # keys?


Jonathan Langevin
Systems Administrator

Loom Inc.
Wilmington, NC: (910) 241-0433 - [hidden email] - www.loomlearning.com - Skype: intel352



On Mon, Jul 4, 2011 at 10:14 AM, Justin Sheehy <[hidden email]> wrote:
Hi, Jonathan.

On Mon, Jul 4, 2011 at 9:42 AM, Jonathan Langevin
<[hidden email]> wrote:

> I've seen users show concern of Bitcask's space usage overhead. How does that
> compare against LevelDB?

Bitcask doesn't have much in the way of disk space "overhead" unless
you mean that the space used by deleted or overwritten values is not
reclaimed until after a merge. In that way LevelDB is similar since
space used by deleted and overwritten items is reclaimed as they are
moved into older "levels" of the DB. The behavior here is not
identical, but similar in concept.

By way of comparison, InnoDB imposes about a 2x space overhead cost on
many common datasets but the overhead is usually fairly static.

> If using a Level backend, what advantages do we lose of Bitcask? ls replication &
> availability an issue at all?

The functionality provided by Riak above the storage engines (such as
replication and system-wide availability) are generally not impacted
by your choice of storage engine.

There are two main things you would lose today:

1 - latency
2 - stability

The first of these is fundamental: for many usage patterns Bitcask
will have a latency advantage over LevelDB due to being able to
guarantee that it will never perform more than a single disk seek per
operation.

The second is just about the relative immaturity of LevelDB: we have
not yet seen LevelDB in production environments for an extended amount
of time as we have with Bitcask. Anyone using it now as a Bitcask
replacement should realize that they are on the leading edge and
taking the usual risks that come with adopting new software. That
said, we expect LevelDB to do well over time as one of the alternative
Riak storage engines.

The main reason to use LevelDB under Riak would be if your number of
keys is huge and thus the RAM consumption of Bitcask would make it
unsuitable. That is, we expect people to use LevelDB in the same
situations that they might previously have chosen Innostore as their
storage engine.

-Justin


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Justin Sheehy
On Mon, Jul 4, 2011 at 10:33 AM, Jonathan Langevin
<[hidden email]> wrote:

> Thanks Justin for the helpful response :-)

Happy to help.

> Can you define what you would consider "huge" regarding # keys?

A bit depends on the details (such as key size) but generally the
tipping point is somewhere near ten million keys per GB of RAM.

-Justin

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Will Moss
Hey Justin/Dave,

I know you guys are busy working on various new features, is this going to make it into the next production release? Is there a date for that? Is there somewhere to look at your road map so I don't have to spam the list?

Thanks,
Will


On Mon, Jul 4, 2011 at 7:41 AM, Justin Sheehy <[hidden email]> wrote:
On Mon, Jul 4, 2011 at 10:33 AM, Jonathan Langevin
<[hidden email]> wrote:

> Thanks Justin for the helpful response :-)

Happy to help.

> Can you define what you would consider "huge" regarding # keys?

A bit depends on the details (such as key size) but generally the
tipping point is somewhere near ten million keys per GB of RAM.

-Justin

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Dave Smith
On Mon, Jul 11, 2011 at 4:24 PM, Will Moss <[hidden email]> wrote:

> I know you guys are busy working on various new features, is this going to
> make it into the next production release? Is there a date for that? Is there
> somewhere to look at your road map so I don't have to spam the list?

Hi Will,

Yes, the current plan is to have LevelDB a default part of the next
production release.

We're still hashing out the timing of the next release, but it will
_probably_ be sometime in the next 3 months (or so...).

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Phil Stanhope
Can you share some LevelDB config examples?

I've built latest from github and know that I've got the elevedb driver/wrapper.

make devrel is producing a {storage_backend, riak_kv_bitcask_backend} ... is it as simple as changing this to riak_kv_eleveldb_backend?

Perhaps I missed it ... but did you guys also share the sample data that you used for the benchmark and instructions about how to attempt recreation of it?

For example, the blog post mentioned "commodity" server. But I don't recall whether a mention was made of a stock 3 or 4 node localhost cluster, where all nodes hit by the benchmark, etc.

This would be excellent information to make available to the community.

-phil

On Tue, Jul 12, 2011 at 11:16 AM, David Smith <[hidden email]> wrote:
On Mon, Jul 11, 2011 at 4:24 PM, Will Moss <[hidden email]> wrote:

> I know you guys are busy working on various new features, is this going to
> make it into the next production release? Is there a date for that? Is there
> somewhere to look at your road map so I don't have to spam the list?

Hi Will,

Yes, the current plan is to have LevelDB a default part of the next
production release.

We're still hashing out the timing of the next release, but it will
_probably_ be sometime in the next 3 months (or so...).

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: LevelDB driver

Justin Sheehy
Hi, Phil.

I might have caused a little confusion. I mentioned, but perhaps
didn't sufficiently emphasize, that the benchmark comparing LevelDB to
InnoDB was not a benchmark of Riak at all, but just directly talking
to the storage engines in order to look at the feasibility of doing
more with LevelDB.

That is why there is no mention of how many nodes or any such thing:
it was a one-machine test of embedded storage engines. The data was
generated by basho_bench during the tests, initially using the
sequential_int_gen generator and then using the pareto generators for
subsequent access.

As LevelDB becomes more fully supported as a backend for Riak, we will
certainly publish directions and examples for configuration.

-Justin

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com