Exporting data from Riak KV in order to let Android read it offline

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Exporting data from Riak KV in order to let Android read it offline

Luca Favatella
Hi All,

What file format easy to read on an Android mobile device would you recommend for representing a snapshot of part of a live Riak KV store? The main aim is minimizing development effort on the Android device while keeping battery consumption reasonable.


As Riak KV can be defined as a key-value store with key being any arbitrary byte sequence and value being any [MIME-typed][1] arbitrary byte sequence, I see that any Android-friendly file format / library for reading any such export needs to:
* Be key-value; and
* Be able to return the value and its MIME type.

[1]: Even values with siblings [can][2] be represented with MIME type.
[2]: http://docs.basho.com/riak/kv/2.1.4/developing/api/http/fetch-object/#get-all-siblings-in-one-request

Some of the alternative file formats include:
* A. SQLite, with columns key (the same as in Riak KV), mime type and value - respectively of types TEXT, TEXT and BLOB;
* B. Archive (e.g. tar, zip) containing file paths corresponding to exported Riak KV keys, with file content being the Riak KV [HTTP GET response][3] - that includes MIME type and body;
* C. Archived (because potentially multi-file) LevelDB database, with the key being the Riak KV key and the value being the Riak KV [HTTP GET response][3] - that includes MIME type and body. (Based on a quick search there is some code for LevelDB on Android.)
Storing the HTTP GET response require the Android application to parse it for extracting the MIME type and the actual value. (I assume there is some HTTP parser Android library).

[3]: http://docs.basho.com/riak/kv/2.1.4/developing/api/http/fetch-object/#simple-example


Feel free to suggest libraries / formats that attempt to provide a "database adapter" on Android by making assumption on the MIME type (e.g. JSON) of the value. Android libraries that attempt to model / re-use similar read API as Riak KV HTTP / Protocol Buffers API are welcome too.


I am aware of https://github.com/basho-labs/riak-data-migrator but it does not seem relevant mainly because:
* The format of the dump on disk (documented as “Protocol Buffer format”) appears not to be documented / meant for reading as a key-value store by other software e.g. Android;
* It lists keys in buckets - that I would prefer to avoid because that operation is expensive and because I would prefer to avoid designing buckets around a fixed exporting policy.

I am aware of https://github.com/couchbase/couchbase-lite-android but it seems too much as a read-only offline export. The mobile device does not need to sync the data: when a new export file is available on the server, the mobile device can fetch the new file and delete the old one.


Thanks and regards
Luca


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Exporting data from Riak KV in order to let Android read it offline

Luca Favatella
On 7 November 2016 at 16:07, Luca Favatella
<[hidden email]> wrote:

>
> Hi All,
>
> What file format easy to read on an Android mobile device would you recommend for representing a snapshot of part of a live Riak KV store? The main aim is minimizing development effort on the Android device while keeping battery consumption reasonable.
>
>
> As Riak KV can be defined as a key-value store with key being any arbitrary byte sequence and value being any [MIME-typed][1] arbitrary byte sequence, I see that any Android-friendly file format / library for reading any such export needs to:
> * Be key-value; and
> * Be able to return the value and its MIME type.
>
> [1]: Even values with siblings [can][2] be represented with MIME type.
> [2]: http://docs.basho.com/riak/kv/2.1.4/developing/api/http/fetch-object/#get-all-siblings-in-one-request
>
> Some of the alternative file formats include:
> * A. SQLite, with columns key (the same as in Riak KV), mime type and value - respectively of types TEXT, TEXT and BLOB;
> * B. Archive (e.g. tar, zip) containing file paths corresponding to exported Riak KV keys, with file content being the Riak KV [HTTP GET response][3] - that includes MIME type and body;
> * C. Archived (because potentially multi-file) LevelDB database, with the key being the Riak KV key and the value being the Riak KV [HTTP GET response][3] - that includes MIME type and body. (Based on a quick search there is some code for LevelDB on Android.)
> Storing the HTTP GET response require the Android application to parse it for extracting the MIME type and the actual value. (I assume there is some HTTP parser Android library).
>
> [3]: http://docs.basho.com/riak/kv/2.1.4/developing/api/http/fetch-object/#simple-example

Hi,

Just an update for letting you know that my current plan is exploiting
the HTTP 1.1 standardized caching semantics (RFC 7234) and an HTTP
cache on Android (initially class `HttpResponseCache`).

The mobile device would send HTTP GET request to a front-end HTTP
service hiding Riak KV (HTTP or PB), and would cache the responses
locally for offline reading.

(This solution requires changes to the functional specifications of
the project I am working on so I had not listed it as an alternative.)

Regards
Luca

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com