Bitcask won't merge without explicit merge() call

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Bitcask won't merge without explicit merge() call

Dmitry Groshev
I'm using standalone Bitcask (no Riak at all). It is used as a storage
backend for an in-memory database, so the load is as follows: every 5
minutes I write ~1.2GB of data (~20k keys ~60kb data each) to the same
keys as before, overwriting their contents. Then I call
bitcask:sync(BitcaskRef). The problem is that if I don't call
bitcask:merge(Path) explicitly, Bitcask won't shrink it's size (and it
shrinks if I call it). Here is an example of what I'm talking about:

after ~15 minutes from the start:

total 4.4G
drwxr-xr-x  2 root root 260K Dec 12 22:51 .
drwxr-xr-x 22 root root 4.0K Dec  5 05:16 ..
-rw-------  1 root root 2.0G Dec 12 22:41 1323714971.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 22:41 1323714971.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 22:51 1323715283.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 22:51 1323715283.bitcask.hint
-rw-------  1 root root 366M Dec 12 22:51 1323715882.bitcask.data
-rw-r--r--  1 root root 286K Dec 12 22:51 1323715882.bitcask.hint
-rw-------  1 root root   45 Dec 12 22:51 bitcask.write.lock

after an hour:

total 19G
drwxr-xr-x  2 root root 260K Dec 12 23:56 .
drwxr-xr-x 22 root root 4.0K Dec  5 05:16 ..
-rw-------  1 root root 2.0G Dec 12 22:41 1323714971.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 22:41 1323714971.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 22:51 1323715283.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 22:51 1323715283.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:01 1323715882.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 23:01 1323715882.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:11 1323716480.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 23:11 1323716480.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:21 1323717078.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 23:21 1323717078.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:31 1323717676.bitcask.data
-rw-r--r--  1 root root 2.3M Dec 12 23:31 1323717676.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:36 1323718271.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 23:36 1323718271.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:46 1323718583.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 23:46 1323718583.bitcask.hint
-rw-------  1 root root 2.0G Dec 12 23:56 1323719182.bitcask.data
-rw-r--r--  1 root root 2.7M Dec 12 23:56 1323719182.bitcask.hint
-rw-------  1 root root 541M Dec 12 23:56 1323719779.bitcask.data
-rw-r--r--  1 root root 422K Dec 12 23:56 1323719779.bitcask.hint
-rw-------  1 root root   45 Dec 12 23:56 bitcask.write.lock

I didn't change any Bitcask settings:

application:get_all_env(bitcask).
[{frag_merge_trigger,60},
 {dead_bytes_merge_trigger,536870912},
 {frag_threshold,40},
 {included_applications,[]},
 {open_timeout,4},
 {dead_bytes_threshold,134217728},
 {max_file_size,2147483648},
 {max_fold_puts,0},
 {max_fold_age,-1},
 {small_file_threshold,10485760},
 {sync_strategy,none},
 {expiry_secs,-1},
 {merge_window,always}]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Bitcask won't merge without explicit merge() call

Justin Sheehy
Dmitry,

What you are expecting is Bitcask's normal behavior, though I can see why it might not be what you expected.

Bitcask does not quite auto-merge; instead it provides you with the tools to easily decide when a merge is needed, and to easily have a merge scheduled when you wish.

Does this example of usage clarify it for you? https://github.com/basho/riak_kv/blob/master/src/riak_kv_bitcask_backend.erl#L371-374

We should probably create better documentation for this aspect of Bitcask usage in any case.

-Justin





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com