raw http example with scary results

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

raw http example with scary results

Richard Bucker
In the online docs for the http raw... there is an example:

and the response is supposed to look something like:

{"props":{"n_val":4,"name":"example","allow_mult":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"raw_link_walker_resource","fun":"mapreduce_linkfun"},"old_vclock":86400,"small_vclock":10,"young_vclock":21600},"keys":["foo"]}

what caught my attention was the "keys":["foo"].  If there were 10K or 10M or more records with unique key values (possibly UUIDs)...... this response string would be HUGE and probably unmanageable.

Is there a better reference for HTTP type functionality?

/r

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: raw http example with scary results

bryan-basho
Administrator
On Wed, Feb 17, 2010 at 9:56 AM, Richard Bucker <[hidden email]> wrote:

> In the online docs for the http raw... there is an example:
>      curl http://127.0.0.1:8098/raw/example
> and the response is supposed to look something like:
>
> {"props":{"n_val":4,"name":"example","allow_mult":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"raw_link_walker_resource","fun":"mapreduce_linkfun"},"old_vclock":86400,"small_vclock":10,"young_vclock":21600},"keys":["foo"]}
>
> what caught my attention was the "keys":["foo"].  If there were 10K or 10M
> or more records with unique key values (possibly UUIDs)...... this response
> string would be HUGE and probably unmanageable.
> Is there a better reference for HTTP type functionality?

Hi, Richard.  The raw resource accepts a "keys" parameter to help with
this case.

The default, equivalent to keys=true, will provide a response with all
the keys in it, like you received above.

keys=false will omit the "keys" field from the response.

keys=chunked will provide the keylist via chunked HTTP encoding.  Each
chunk will have a portion of the keylist, {"keys":["x","y","z",...]},
so the client may consume it a little bit at a time.

Does that solve the issue for your use case?

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: raw http example with scary results

Richard Bucker
Good answer... if I were the DOCs where would I be hiding? I'm sure there are plenty of flags etc that I might like to be familiar with?

/r

On Wed, Feb 17, 2010 at 10:23 AM, Bryan Fink <[hidden email]> wrote:
On Wed, Feb 17, 2010 at 9:56 AM, Richard Bucker <[hidden email]> wrote:
> In the online docs for the http raw... there is an example:
>      curl http://127.0.0.1:8098/raw/example
> and the response is supposed to look something like:
>
> {"props":{"n_val":4,"name":"example","allow_mult":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"raw_link_walker_resource","fun":"mapreduce_linkfun"},"old_vclock":86400,"small_vclock":10,"young_vclock":21600},"keys":["foo"]}
>
> what caught my attention was the "keys":["foo"].  If there were 10K or 10M
> or more records with unique key values (possibly UUIDs)...... this response
> string would be HUGE and probably unmanageable.
> Is there a better reference for HTTP type functionality?

Hi, Richard.  The raw resource accepts a "keys" parameter to help with
this case.

The default, equivalent to keys=true, will provide a response with all
the keys in it, like you received above.

keys=false will omit the "keys" field from the response.

keys=chunked will provide the keylist via chunked HTTP encoding.  Each
chunk will have a portion of the keylist, {"keys":["x","y","z",...]},
so the client may consume it a little bit at a time.

Does that solve the issue for your use case?

-Bryan


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: raw http example with scary results

bryan-basho
Administrator
We are in the process of writing fresh docs for this interface and
others.  For now, the most complete documentation of the raw http
interface is in the comments at the top of raw_http_resource.erl.

http://bitbucket.org/basho/riak/src/tip/apps/riak/src/raw_http_resource.erl

Though, I notice that it doesn't mention "keys=chunked" yet.  I'll get
that fixed.  :)

-Bryan


On Wed, Feb 17, 2010 at 10:30 AM, Richard Bucker <[hidden email]> wrote:

> Good answer... if I were the DOCs where would I be hiding? I'm sure there
> are plenty of flags etc that I might like to be familiar with?
> /r
>
> On Wed, Feb 17, 2010 at 10:23 AM, Bryan Fink <[hidden email]> wrote:
>>
>> On Wed, Feb 17, 2010 at 9:56 AM, Richard Bucker <[hidden email]>
>> wrote:
>> > In the online docs for the http raw... there is an example:
>> >      curl http://127.0.0.1:8098/raw/example
>> > and the response is supposed to look something like:
>> >
>> >
>> > {"props":{"n_val":4,"name":"example","allow_mult":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_util","fun":"chash_std_keyfun"},"linkfun":{"mod":"raw_link_walker_resource","fun":"mapreduce_linkfun"},"old_vclock":86400,"small_vclock":10,"young_vclock":21600},"keys":["foo"]}
>> >
>> > what caught my attention was the "keys":["foo"].  If there were 10K or
>> > 10M
>> > or more records with unique key values (possibly UUIDs)...... this
>> > response
>> > string would be HUGE and probably unmanageable.
>> > Is there a better reference for HTTP type functionality?
>>
>> Hi, Richard.  The raw resource accepts a "keys" parameter to help with
>> this case.
>>
>> The default, equivalent to keys=true, will provide a response with all
>> the keys in it, like you received above.
>>
>> keys=false will omit the "keys" field from the response.
>>
>> keys=chunked will provide the keylist via chunked HTTP encoding.  Each
>> chunk will have a portion of the keylist, {"keys":["x","y","z",...]},
>> so the client may consume it a little bit at a time.
>>
>> Does that solve the issue for your use case?
>>
>> -Bryan
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com