DETS vs Innostore

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

DETS vs Innostore

Abeming Sf
Hi,

I did some testing between dets and innostore for storage backend which store 200k of data with same structure. Using default configuration for both storage backend, running on singgle node and bucket's allow_multi set to false.

Data example : [{field, "value"}, {field, "value"}, {field, "value"}]

I found out that storing 200k of data sequentially into dets is a lot more faster than storing into innodb and dets only use around 350MB space while innodb use around 1.1GB.

Also, in my other test which store 1 million data into dets, it only use around 700MB++ of space (same data structure above).

It is recommended to use innostore for production right?What should I do to make it better.

Regards,
Abe

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
wde
Reply | Threaded
Open this post in threaded view
|

Re: DETS vs Innostore

wde
I never used innostore, but dets seems to be a good solution if you don't have to store more than 2Gb in one file :)


>Hi,
>
>I did some testing between dets and innostore for storage backend which
>store 200k of data with same structure. Using default configuration for both
>storage backend, running on singgle node and bucket's allow_multi set to
>false.
>
>Data example : [{field, "value"}, {field, "value"}, {field, "value"}]
>
>I found out that storing 200k of data sequentially into dets is a lot more
>faster than storing into innodb and dets only use around 350MB space while
>innodb use around 1.1GB.
>
>Also, in my other test which store 1 million data into dets, it only use
>around 700MB++ of space (same data structure above).
>
>It is recommended to use innostore for production right?What should I do to
>make it better.
>
>Regards,
>Abe
>
>_______________________________________________
>riak-users mailing list
>[hidden email]
>http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: DETS vs Innostore

Dave Smith
In reply to this post by Abeming Sf
Hi Abe,

On Mon, Apr 26, 2010 at 11:31 PM, Abeming Sf <[hidden email]> wrote:
Hi,

Data example : [{field, "value"}, {field, "value"}, {field, "value"}]

First observation -- that's only ~61 bytes of data  when encoded into binary -- which is pretty small. If you start storing larger chunks of data (say 10k), you'll find that dets starts running into significant iowait issues as it tries to flush the data to disk.
 
I found out that storing 200k of data sequentially into dets is a lot more faster than storing into innodb and dets only use around 350MB space while innodb use around 1.1GB.

Yes, inno does introduce expansion, particularly when you are using very small piece of data. I believe this has something to do with the way that Inno allocates a 8 KB page per BLOB. 
 
Also, in my other test which store 1 million data into dets, it only use around 700MB++ of space (same data structure above).

It is recommended to use innostore for production right?What should I do to make it better.

Whether or not you should use innostore vs. dets in production depends heavily on what your access pattern is and how much data you plan to store. If you've going to be storing many gigabytes of data, innostore will do much better than dets as the dataset grows large. Innostore has better durability mechanisms that dets as well, so that's something to consider too.

Dets is very fast in situations where you have a small-ish amount of data and reads predominate your access pattern. 

Hope that helps.

D.

 

Regards,
Abe

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: DETS vs Innostore

Abeming Sf
Hi Dave and wde,

Thanks for ur explanation, it helps me to decide which storage backend should I use. Appreciate that.

Regards,
Abe

On 28 April 2010 20:48, David Smith <[hidden email]> wrote:
Hi Abe,

On Mon, Apr 26, 2010 at 11:31 PM, Abeming Sf <[hidden email]> wrote:
Hi,


Data example : [{field, "value"}, {field, "value"}, {field, "value"}]

First observation -- that's only ~61 bytes of data  when encoded into binary -- which is pretty small. If you start storing larger chunks of data (say 10k), you'll find that dets starts running into significant iowait issues as it tries to flush the data to disk.
 
I found out that storing 200k of data sequentially into dets is a lot more faster than storing into innodb and dets only use around 350MB space while innodb use around 1.1GB.

Yes, inno does introduce expansion, particularly when you are using very small piece of data. I believe this has something to do with the way that Inno allocates a 8 KB page per BLOB. 
 
Also, in my other test which store 1 million data into dets, it only use around 700MB++ of space (same data structure above).

It is recommended to use innostore for production right?What should I do to make it better.

Whether or not you should use innostore vs. dets in production depends heavily on what your access pattern is and how much data you plan to store. If you've going to be storing many gigabytes of data, innostore will do much better than dets as the dataset grows large. Innostore has better durability mechanisms that dets as well, so that's something to consider too.

Dets is very fast in situations where you have a small-ish amount of data and reads predominate your access pattern. 

Hope that helps.

D.

 

Regards,
Abe

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: DETS vs Innostore

Abeming Sf
Sorry it's David...not Dave...:)

On 29 April 2010 00:32, Abeming Sf <[hidden email]> wrote:
Hi Dave and wde,

Thanks for ur explanation, it helps me to decide which storage backend should I use. Appreciate that.

Regards,
Abe


On 28 April 2010 20:48, David Smith <[hidden email]> wrote:
Hi Abe,

On Mon, Apr 26, 2010 at 11:31 PM, Abeming Sf <[hidden email]> wrote:
Hi,


Data example : [{field, "value"}, {field, "value"}, {field, "value"}]

First observation -- that's only ~61 bytes of data  when encoded into binary -- which is pretty small. If you start storing larger chunks of data (say 10k), you'll find that dets starts running into significant iowait issues as it tries to flush the data to disk.
 
I found out that storing 200k of data sequentially into dets is a lot more faster than storing into innodb and dets only use around 350MB space while innodb use around 1.1GB.

Yes, inno does introduce expansion, particularly when you are using very small piece of data. I believe this has something to do with the way that Inno allocates a 8 KB page per BLOB. 
 
Also, in my other test which store 1 million data into dets, it only use around 700MB++ of space (same data structure above).

It is recommended to use innostore for production right?What should I do to make it better.

Whether or not you should use innostore vs. dets in production depends heavily on what your access pattern is and how much data you plan to store. If you've going to be storing many gigabytes of data, innostore will do much better than dets as the dataset grows large. Innostore has better durability mechanisms that dets as well, so that's something to consider too.

Dets is very fast in situations where you have a small-ish amount of data and reads predominate your access pattern. 

Hope that helps.

D.

 

Regards,
Abe

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com