Riak for streaming video

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak for streaming video

Alex De la rosa
Hi there,

Imagine that I want to use Riak for a video service like Youtube and I save the video file in Riak... is there a way to stream the contents of 1 key? I know we can stream keys... but what about its content?

It would be pretty troublesome to have to wait for the full data to be downloaded $bucket->get('myvideo') to be able to serve it... as they can be pretty big files.

Is there any recommendation to store/get big files like videos? or it would be better to use some other system than Riak for the job?

Thanks,
Alex

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Hector Castro-2
Hi Alex,

For this type of problem, you may want to look into Riak CS. [0] It is
an object storage platform built on top of Riak.

When you GET an object from Riak CS, you can specify a Range header to
get the object by its byte offset. [1]

--
Hector

[0] http://docs.basho.com/riakcs/latest/
[1] http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples

On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
<[hidden email]> wrote:

> Hi there,
>
> Imagine that I want to use Riak for a video service like Youtube and I save
> the video file in Riak... is there a way to stream the contents of 1 key? I
> know we can stream keys... but what about its content?
>
> It would be pretty troublesome to have to wait for the full data to be
> downloaded $bucket->get('myvideo') to be able to serve it... as they can be
> pretty big files.
>
> Is there any recommendation to store/get big files like videos? or it would
> be better to use some other system than Riak for the job?
>
> Thanks,
> Alex
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Alex De la rosa
Hi Hector,

I see, I always thought that Riak CS was the same as Riak just that you run it in Amazon S3... they are actually different...

So... if I have a social network and one of the features is that they can share video, I would use a normal Riak cluster for the webapp and data and a Riak CS cluster for the video storage/streaming... am I right?

By the way, couldn't this "range header" be implemented in standard Riak? might be a good thing to have so you don't need 2 clusters for this matter.

Thanks,
Alex


On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro <[hidden email]> wrote:
Hi Alex,

For this type of problem, you may want to look into Riak CS. [0] It is
an object storage platform built on top of Riak.

When you GET an object from Riak CS, you can specify a Range header to
get the object by its byte offset. [1]

--
Hector

[0] http://docs.basho.com/riakcs/latest/
[1] http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples

On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
<[hidden email]> wrote:
> Hi there,
>
> Imagine that I want to use Riak for a video service like Youtube and I save
> the video file in Riak... is there a way to stream the contents of 1 key? I
> know we can stream keys... but what about its content?
>
> It would be pretty troublesome to have to wait for the full data to be
> downloaded $bucket->get('myvideo') to be able to serve it... as they can be
> pretty big files.
>
> Is there any recommendation to store/get big files like videos? or it would
> be better to use some other system than Riak for the job?
>
> Thanks,
> Alex
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Hector Castro-2
Hey Alex,

My response are inline below.

--
Hector


On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
<[hidden email]> wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.

> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).

> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.

> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro <[hidden email]> wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> <[hidden email]> wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Alex De la rosa
Hi Hector,

Thanks for your response :) highly appreciated. I have more questions though.

Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?

What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.

Thanks,
Alex


On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro <[hidden email]> wrote:
Hey Alex,

My response are inline below.

--
Hector


On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
<[hidden email]> wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.

> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).

> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.

> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro <[hidden email]> wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> <[hidden email]> wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Jason Campbell-2
Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <[hidden email]>
To: "Hector Castro" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video



Hi Hector,


Thanks for your response :) highly appreciated. I have more questions though.


Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?


What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.


Thanks,
Alex



On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] > wrote:


Hey Alex,

My response are inline below.

--
Hector



On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
< [hidden email] > wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.


> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).


> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.



> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] > wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/ 
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples 
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> < [hidden email] > wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com 
>> >
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Jared Morrow
Luwak is definitely EOL'd and RiakCS is our large object store going forward.  It is a far superior design compared to Luwak and handles very large file sizes. 



On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;alex.rosa.box@gmail.com&#39;)">alex.rosa.box@...>
To: "Hector Castro" <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;hector@basho.com&#39;)">hector@...>
Cc: "riak-users" <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video



Hi Hector,


Thanks for your response :) highly appreciated. I have more questions though.


Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?


What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.


Thanks,
Alex



On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;hector@basho.com&#39;)">hector@... > wrote:


Hey Alex,

My response are inline below.

--
Hector



On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
< <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;alex.rosa.box@gmail.com&#39;)">alex.rosa.box@... > wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.


> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).


> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.



> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;hector@basho.com&#39;)">hector@... > wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> < <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;alex.rosa.box@gmail.com&#39;)">alex.rosa.box@... > wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > <a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>


_______________________________________________
riak-users mailing list
<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;riak-users@lists.basho.com&#39;)">riak-users@...
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Alex De la rosa
Hi there,

One more question regarding Riak CS, does it behave like standard Riak? storing 3 copies (default) and needing a minimum of 5 servers to operate?

Cheers,
Alex


On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <[hidden email]> wrote:
Luwak is definitely EOL'd and RiakCS is our large object store going forward.  It is a far superior design compared to Luwak and handles very large file sizes. 



On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <[hidden email]>
To: "Hector Castro" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video



Hi Hector,


Thanks for your response :) highly appreciated. I have more questions though.


Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?


What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.


Thanks,
Alex



On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] > wrote:


Hey Alex,

My response are inline below.

--
Hector



On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
< [hidden email] > wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.


> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).


> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.



> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] > wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> < [hidden email] > wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Jared Morrow
Alex,

That is correct.  There are some additional docs we have on load-balancing and such on our docs page http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/, but generally it acts and feels like a Riak installation.  One additional special node is required (Stanchion) and that is also described on our docs page http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/.

Keep the questions coming, if I don't know the details I'll hand off to Engineers more knowledgable with RiakCS.

-Jared




On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <[hidden email]> wrote:
Hi there,

One more question regarding Riak CS, does it behave like standard Riak? storing 3 copies (default) and needing a minimum of 5 servers to operate?

Cheers,
Alex


On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <[hidden email]> wrote:
Luwak is definitely EOL'd and RiakCS is our large object store going forward.  It is a far superior design compared to Luwak and handles very large file sizes. 



On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <[hidden email]>
To: "Hector Castro" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video



Hi Hector,


Thanks for your response :) highly appreciated. I have more questions though.


Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?


What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.


Thanks,
Alex



On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] > wrote:


Hey Alex,

My response are inline below.

--
Hector



On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
< [hidden email] > wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.


> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).


> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.



> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] > wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> < [hidden email] > wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Alex De la rosa
Hi Jared,

Thanks for your explanation and links... my next question is... Is Stanchion not a single-point of failure? One of the things I like in Standard Riak is that every node has the same level, there are no masters, no slaves... if one fail it doesn't matter, the rest can do the job. If Riak CS requires un especial node, what happens if that one fails? system is down then? Can you configure the system with more than one Stanchion node?

Cheers,
Alex


On Mon, Jun 30, 2014 at 7:37 AM, Jared Morrow <[hidden email]> wrote:
Alex,

That is correct.  There are some additional docs we have on load-balancing and such on our docs page http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/, but generally it acts and feels like a Riak installation.  One additional special node is required (Stanchion) and that is also described on our docs page http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/.

Keep the questions coming, if I don't know the details I'll hand off to Engineers more knowledgable with RiakCS.

-Jared




On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <[hidden email]> wrote:
Hi there,

One more question regarding Riak CS, does it behave like standard Riak? storing 3 copies (default) and needing a minimum of 5 servers to operate?

Cheers,
Alex


On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <[hidden email]> wrote:
Luwak is definitely EOL'd and RiakCS is our large object store going forward.  It is a far superior design compared to Luwak and handles very large file sizes. 



On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <[hidden email]>
To: "Hector Castro" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video



Hi Hector,


Thanks for your response :) highly appreciated. I have more questions though.


Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?


What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.


Thanks,
Alex



On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] > wrote:


Hey Alex,

My response are inline below.

--
Hector



On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
< [hidden email] > wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.


> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).


> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.



> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] > wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> < [hidden email] > wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Brian Akins
Stanchion is only used for user modification and creating/deleting buckets. The cluster will function just fine without it except for those functions.  


On Jun 30, 2014, at 3:10 AM, Alex De la rosa <[hidden email]> wrote:

Hi Jared,

Thanks for your explanation and links... my next question is... Is Stanchion not a single-point of failure? One of the things I like in Standard Riak is that every node has the same level, there are no masters, no slaves... if one fail it doesn't matter, the rest can do the job. If Riak CS requires un especial node, what happens if that one fails? system is down then? Can you configure the system with more than one Stanchion node?

Cheers,
Alex


On Mon, Jun 30, 2014 at 7:37 AM, Jared Morrow <[hidden email]> wrote:
Alex,

That is correct.  There are some additional docs we have on load-balancing and such on our docs page http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/, but generally it acts and feels like a Riak installation.  One additional special node is required (Stanchion) and that is also described on our docs page http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/.

Keep the questions coming, if I don't know the details I'll hand off to Engineers more knowledgable with RiakCS.

-Jared




On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <[hidden email]> wrote:
Hi there,

One more question regarding Riak CS, does it behave like standard Riak? storing 3 copies (default) and needing a minimum of 5 servers to operate?

Cheers,
Alex


On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <[hidden email]> wrote:
Luwak is definitely EOL'd and RiakCS is our large object store going forward.  It is a far superior design compared to Luwak and handles very large file sizes. 



On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
Riak CS is designed to be the same interface as Amazon S3.  It can be run anywhere you can run Riak.

Think of it as a way to run your own S3.

I think the hard limit on Riak objects is 50MB, but the recommended size is less than 1MB per object.  The main reason for that is latency.  It's faster to get 100x1MB chunks from the entire cluster than it is to get 1x100MB chunk from a single server.  It's also easier for disks to pull a 1MB object without delaying any other requests than pulling a 100MB object, which may end up queuing other requests made after.

There used to be a project called luwak that was built into Riak and handled chunking files like this, but it has been EOL'd.  Perhaps someone can shed some light on this.  Is RiakCS the final solution for binary storage?  Or is there another way to store larger objects in Riak?

RiakCS is wonderful, but it does seem overkill for some uses where you don't need to expose an S3 interface, don't need the S3 permission structure, and really just want file storage.

----- Original Message -----
From: "Alex De la rosa" <[hidden email]>
To: "Hector Castro" <[hidden email]>
Cc: "riak-users" <[hidden email]>
Sent: Friday, 27 June, 2014 6:55:13 AM
Subject: Re: Riak for streaming video



Hi Hector,


Thanks for your response :) highly appreciated. I have more questions though.


Riak CS has to be implemented in Amazon S3? Or can I house it myself with own managed servers?


What is the limit for an object in Standard Riak? Large objects should go to Riak CS, but what would be a limit for the Standard Riak? I may want to save pictures, for example, in binary data... not a RAW version, but like processed photo uploads from users... I think Standard Riak can hold that with no problem, but it would be good to know the size limit for a key's content.


Thanks,
Alex



On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] > wrote:


Hey Alex,

My response are inline below.

--
Hector



On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
< [hidden email] > wrote:
> Hi Hector,
>
> I see, I always thought that Riak CS was the same as Riak just that you run
> it in Amazon S3... they are actually different...

Correct, but it is important to note that Riak is still at the core of Riak CS.


> So... if I have a social network and one of the features is that they can
> share video, I would use a normal Riak cluster for the webapp and data and a
> Riak CS cluster for the video storage/streaming... am I right?

That seems like a reasonable conclusion.

Riak is fundamentally a distributed key/value store for low latency
access to smaller pieces of data. Riak CS is an S3/Swift
API-compatible object storage platform with a little higher request
latency, but also the added ability to house very large objects
(backups, raw images, video).


> By the way, couldn't this "range header" be implemented in standard Riak?
> might be a good thing to have so you don't need 2 clusters for this matter.

It could some day. Right now, separating the clusters is best because
tuning a single cluster for both use cases would be difficult.



> Thanks,
> Alex
>
>
> On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] > wrote:
>>
>> Hi Alex,
>>
>> For this type of problem, you may want to look into Riak CS. [0] It is
>> an object storage platform built on top of Riak.
>>
>> When you GET an object from Riak CS, you can specify a Range header to
>> get the object by its byte offset. [1]
>>
>> --
>> Hector
>>
>> [0] http://docs.basho.com/riakcs/latest/
>> [1]
>> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>
>> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>> < [hidden email] > wrote:
>> > Hi there,
>> >
>> > Imagine that I want to use Riak for a video service like Youtube and I
>> > save
>> > the video file in Riak... is there a way to stream the contents of 1
>> > key? I
>> > know we can stream keys... but what about its content?
>> >
>> > It would be pretty troublesome to have to wait for the full data to be
>> > downloaded $bucket->get('myvideo') to be able to serve it... as they can
>> > be
>> > pretty big files.
>> >
>> > Is there any recommendation to store/get big files like videos? or it
>> > would
>> > be better to use some other system than Riak for the job?
>> >
>> > Thanks,
>> > Alex
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Kota Uenishi
There are a lot of variety, in definition of being SPOF. It also
depends on not only software implementation but also on setup
configuration and operation.

When it comes to Stanchion, it just serializes creation, modification
and deletion of buckets and users, and all data is stored in Riak,
which means, Stanchion failure does not cause any data loss but just
downtime of these operations. As Brian points out that all other
operations on objects will work fine.

Even you can run two Act-Standby Stanchion processes at once. But the
only thing operators should not do is, to let any two CS nodes see
different Stanchion at once. After the operator make sure 'primary'
Stanchion killed or stopped, it's okay to do any style of failover.

On Mon, Jun 30, 2014 at 7:27 PM, Brian Akins <[hidden email]> wrote:

> Stanchion is only used for user modification and creating/deleting buckets.
> The cluster will function just fine without it except for those functions.
>
>
> On Jun 30, 2014, at 3:10 AM, Alex De la rosa <[hidden email]>
> wrote:
>
> Hi Jared,
>
> Thanks for your explanation and links... my next question is... Is Stanchion
> not a single-point of failure? One of the things I like in Standard Riak is
> that every node has the same level, there are no masters, no slaves... if
> one fail it doesn't matter, the rest can do the job. If Riak CS requires un
> especial node, what happens if that one fails? system is down then? Can you
> configure the system with more than one Stanchion node?
>
> Cheers,
> Alex
>
>
> On Mon, Jun 30, 2014 at 7:37 AM, Jared Morrow <[hidden email]> wrote:
>>
>> Alex,
>>
>> That is correct.  There are some additional docs we have on load-balancing
>> and such on our docs page
>> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/,
>> but generally it acts and feels like a Riak installation.  One additional
>> special node is required (Stanchion) and that is also described on our docs
>> page
>> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/.
>>
>> Keep the questions coming, if I don't know the details I'll hand off to
>> Engineers more knowledgable with RiakCS.
>>
>> -Jared
>>
>>
>>
>>
>> On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <[hidden email]>
>> wrote:
>>>
>>> Hi there,
>>>
>>> One more question regarding Riak CS, does it behave like standard Riak?
>>> storing 3 copies (default) and needing a minimum of 5 servers to operate?
>>>
>>> Cheers,
>>> Alex
>>>
>>>
>>> On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <[hidden email]> wrote:
>>>>
>>>> Luwak is definitely EOL'd and RiakCS is our large object store going
>>>> forward.  It is a far superior design compared to Luwak and handles very
>>>> large file sizes.
>>>>
>>>>
>>>>
>>>> On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
>>>>>
>>>>> Riak CS is designed to be the same interface as Amazon S3.  It can be
>>>>> run anywhere you can run Riak.
>>>>>
>>>>> Think of it as a way to run your own S3.
>>>>>
>>>>> I think the hard limit on Riak objects is 50MB, but the recommended
>>>>> size is less than 1MB per object.  The main reason for that is latency.
>>>>> It's faster to get 100x1MB chunks from the entire cluster than it is to get
>>>>> 1x100MB chunk from a single server.  It's also easier for disks to pull a
>>>>> 1MB object without delaying any other requests than pulling a 100MB object,
>>>>> which may end up queuing other requests made after.
>>>>>
>>>>> There used to be a project called luwak that was built into Riak and
>>>>> handled chunking files like this, but it has been EOL'd.  Perhaps someone
>>>>> can shed some light on this.  Is RiakCS the final solution for binary
>>>>> storage?  Or is there another way to store larger objects in Riak?
>>>>>
>>>>> RiakCS is wonderful, but it does seem overkill for some uses where you
>>>>> don't need to expose an S3 interface, don't need the S3 permission
>>>>> structure, and really just want file storage.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Alex De la rosa" <[hidden email]>
>>>>> To: "Hector Castro" <[hidden email]>
>>>>> Cc: "riak-users" <[hidden email]>
>>>>> Sent: Friday, 27 June, 2014 6:55:13 AM
>>>>> Subject: Re: Riak for streaming video
>>>>>
>>>>>
>>>>>
>>>>> Hi Hector,
>>>>>
>>>>>
>>>>> Thanks for your response :) highly appreciated. I have more questions
>>>>> though.
>>>>>
>>>>>
>>>>> Riak CS has to be implemented in Amazon S3? Or can I house it myself
>>>>> with own managed servers?
>>>>>
>>>>>
>>>>> What is the limit for an object in Standard Riak? Large objects should
>>>>> go to Riak CS, but what would be a limit for the Standard Riak? I may want
>>>>> to save pictures, for example, in binary data... not a RAW version, but like
>>>>> processed photo uploads from users... I think Standard Riak can hold that
>>>>> with no problem, but it would be good to know the size limit for a key's
>>>>> content.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] >
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hey Alex,
>>>>>
>>>>> My response are inline below.
>>>>>
>>>>> --
>>>>> Hector
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
>>>>> < [hidden email] > wrote:
>>>>> > Hi Hector,
>>>>> >
>>>>> > I see, I always thought that Riak CS was the same as Riak just that
>>>>> > you run
>>>>> > it in Amazon S3... they are actually different...
>>>>>
>>>>> Correct, but it is important to note that Riak is still at the core of
>>>>> Riak CS.
>>>>>
>>>>>
>>>>> > So... if I have a social network and one of the features is that they
>>>>> > can
>>>>> > share video, I would use a normal Riak cluster for the webapp and
>>>>> > data and a
>>>>> > Riak CS cluster for the video storage/streaming... am I right?
>>>>>
>>>>> That seems like a reasonable conclusion.
>>>>>
>>>>> Riak is fundamentally a distributed key/value store for low latency
>>>>> access to smaller pieces of data. Riak CS is an S3/Swift
>>>>> API-compatible object storage platform with a little higher request
>>>>> latency, but also the added ability to house very large objects
>>>>> (backups, raw images, video).
>>>>>
>>>>>
>>>>> > By the way, couldn't this "range header" be implemented in standard
>>>>> > Riak?
>>>>> > might be a good thing to have so you don't need 2 clusters for this
>>>>> > matter.
>>>>>
>>>>> It could some day. Right now, separating the clusters is best because
>>>>> tuning a single cluster for both use cases would be difficult.
>>>>>
>>>>>
>>>>>
>>>>> > Thanks,
>>>>> > Alex
>>>>> >
>>>>> >
>>>>> > On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] >
>>>>> > wrote:
>>>>> >>
>>>>> >> Hi Alex,
>>>>> >>
>>>>> >> For this type of problem, you may want to look into Riak CS. [0] It
>>>>> >> is
>>>>> >> an object storage platform built on top of Riak.
>>>>> >>
>>>>> >> When you GET an object from Riak CS, you can specify a Range header
>>>>> >> to
>>>>> >> get the object by its byte offset. [1]
>>>>> >>
>>>>> >> --
>>>>> >> Hector
>>>>> >>
>>>>> >> [0] http://docs.basho.com/riakcs/latest/
>>>>> >> [1]
>>>>> >>
>>>>> >> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>>>> >>
>>>>> >> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>>>>> >> < [hidden email] > wrote:
>>>>> >> > Hi there,
>>>>> >> >
>>>>> >> > Imagine that I want to use Riak for a video service like Youtube
>>>>> >> > and I
>>>>> >> > save
>>>>> >> > the video file in Riak... is there a way to stream the contents of
>>>>> >> > 1
>>>>> >> > key? I
>>>>> >> > know we can stream keys... but what about its content?
>>>>> >> >
>>>>> >> > It would be pretty troublesome to have to wait for the full data
>>>>> >> > to be
>>>>> >> > downloaded $bucket->get('myvideo') to be able to serve it... as
>>>>> >> > they can
>>>>> >> > be
>>>>> >> > pretty big files.
>>>>> >> >
>>>>> >> > Is there any recommendation to store/get big files like videos? or
>>>>> >> > it
>>>>> >> > would
>>>>> >> > be better to use some other system than Riak for the job?
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> > Alex
>>>>> >> >
>>>>> >> > _______________________________________________
>>>>> >> > riak-users mailing list
>>>>> >> > [hidden email]
>>>>> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [hidden email]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [hidden email]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



--
Kota UENISHI / @kuenishi
Basho Japan KK

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak for streaming video

Alex De la rosa
Hi Kota,

返事をありがとうございました! :)

Ok, understood, so 1 Stanchion server shouldn't be a problem at all, good to know :)

Cheers,
Alex


On Mon, Jun 30, 2014 at 5:03 PM, Kota Uenishi <[hidden email]> wrote:
There are a lot of variety, in definition of being SPOF. It also
depends on not only software implementation but also on setup
configuration and operation.

When it comes to Stanchion, it just serializes creation, modification
and deletion of buckets and users, and all data is stored in Riak,
which means, Stanchion failure does not cause any data loss but just
downtime of these operations. As Brian points out that all other
operations on objects will work fine.

Even you can run two Act-Standby Stanchion processes at once. But the
only thing operators should not do is, to let any two CS nodes see
different Stanchion at once. After the operator make sure 'primary'
Stanchion killed or stopped, it's okay to do any style of failover.

On Mon, Jun 30, 2014 at 7:27 PM, Brian Akins <[hidden email]> wrote:
> Stanchion is only used for user modification and creating/deleting buckets.
> The cluster will function just fine without it except for those functions.
>
>
> On Jun 30, 2014, at 3:10 AM, Alex De la rosa <[hidden email]>
> wrote:
>
> Hi Jared,
>
> Thanks for your explanation and links... my next question is... Is Stanchion
> not a single-point of failure? One of the things I like in Standard Riak is
> that every node has the same level, there are no masters, no slaves... if
> one fail it doesn't matter, the rest can do the job. If Riak CS requires un
> especial node, what happens if that one fails? system is down then? Can you
> configure the system with more than one Stanchion node?
>
> Cheers,
> Alex
>
>
> On Mon, Jun 30, 2014 at 7:37 AM, Jared Morrow <[hidden email]> wrote:
>>
>> Alex,
>>
>> That is correct.  There are some additional docs we have on load-balancing
>> and such on our docs page
>> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Load-Balancing-and-Proxy-Configuration/,
>> but generally it acts and feels like a Riak installation.  One additional
>> special node is required (Stanchion) and that is also described on our docs
>> page
>> http://docs.basho.com/riakcs/latest/cookbooks/configuration/Configuring-Stanchion/.
>>
>> Keep the questions coming, if I don't know the details I'll hand off to
>> Engineers more knowledgable with RiakCS.
>>
>> -Jared
>>
>>
>>
>>
>> On Sun, Jun 29, 2014 at 5:58 AM, Alex De la rosa <[hidden email]>
>> wrote:
>>>
>>> Hi there,
>>>
>>> One more question regarding Riak CS, does it behave like standard Riak?
>>> storing 3 copies (default) and needing a minimum of 5 servers to operate?
>>>
>>> Cheers,
>>> Alex
>>>
>>>
>>> On Fri, Jun 27, 2014 at 2:43 AM, Jared Morrow <[hidden email]> wrote:
>>>>
>>>> Luwak is definitely EOL'd and RiakCS is our large object store going
>>>> forward.  It is a far superior design compared to Luwak and handles very
>>>> large file sizes.
>>>>
>>>>
>>>>
>>>> On Thursday, June 26, 2014, Jason Campbell <[hidden email]> wrote:
>>>>>
>>>>> Riak CS is designed to be the same interface as Amazon S3.  It can be
>>>>> run anywhere you can run Riak.
>>>>>
>>>>> Think of it as a way to run your own S3.
>>>>>
>>>>> I think the hard limit on Riak objects is 50MB, but the recommended
>>>>> size is less than 1MB per object.  The main reason for that is latency.
>>>>> It's faster to get 100x1MB chunks from the entire cluster than it is to get
>>>>> 1x100MB chunk from a single server.  It's also easier for disks to pull a
>>>>> 1MB object without delaying any other requests than pulling a 100MB object,
>>>>> which may end up queuing other requests made after.
>>>>>
>>>>> There used to be a project called luwak that was built into Riak and
>>>>> handled chunking files like this, but it has been EOL'd.  Perhaps someone
>>>>> can shed some light on this.  Is RiakCS the final solution for binary
>>>>> storage?  Or is there another way to store larger objects in Riak?
>>>>>
>>>>> RiakCS is wonderful, but it does seem overkill for some uses where you
>>>>> don't need to expose an S3 interface, don't need the S3 permission
>>>>> structure, and really just want file storage.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Alex De la rosa" <[hidden email]>
>>>>> To: "Hector Castro" <[hidden email]>
>>>>> Cc: "riak-users" <[hidden email]>
>>>>> Sent: Friday, 27 June, 2014 6:55:13 AM
>>>>> Subject: Re: Riak for streaming video
>>>>>
>>>>>
>>>>>
>>>>> Hi Hector,
>>>>>
>>>>>
>>>>> Thanks for your response :) highly appreciated. I have more questions
>>>>> though.
>>>>>
>>>>>
>>>>> Riak CS has to be implemented in Amazon S3? Or can I house it myself
>>>>> with own managed servers?
>>>>>
>>>>>
>>>>> What is the limit for an object in Standard Riak? Large objects should
>>>>> go to Riak CS, but what would be a limit for the Standard Riak? I may want
>>>>> to save pictures, for example, in binary data... not a RAW version, but like
>>>>> processed photo uploads from users... I think Standard Riak can hold that
>>>>> with no problem, but it would be good to know the size limit for a key's
>>>>> content.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Alex
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 26, 2014 at 10:45 PM, Hector Castro < [hidden email] >
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hey Alex,
>>>>>
>>>>> My response are inline below.
>>>>>
>>>>> --
>>>>> Hector
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jun 26, 2014 at 4:26 PM, Alex De la rosa
>>>>> < [hidden email] > wrote:
>>>>> > Hi Hector,
>>>>> >
>>>>> > I see, I always thought that Riak CS was the same as Riak just that
>>>>> > you run
>>>>> > it in Amazon S3... they are actually different...
>>>>>
>>>>> Correct, but it is important to note that Riak is still at the core of
>>>>> Riak CS.
>>>>>
>>>>>
>>>>> > So... if I have a social network and one of the features is that they
>>>>> > can
>>>>> > share video, I would use a normal Riak cluster for the webapp and
>>>>> > data and a
>>>>> > Riak CS cluster for the video storage/streaming... am I right?
>>>>>
>>>>> That seems like a reasonable conclusion.
>>>>>
>>>>> Riak is fundamentally a distributed key/value store for low latency
>>>>> access to smaller pieces of data. Riak CS is an S3/Swift
>>>>> API-compatible object storage platform with a little higher request
>>>>> latency, but also the added ability to house very large objects
>>>>> (backups, raw images, video).
>>>>>
>>>>>
>>>>> > By the way, couldn't this "range header" be implemented in standard
>>>>> > Riak?
>>>>> > might be a good thing to have so you don't need 2 clusters for this
>>>>> > matter.
>>>>>
>>>>> It could some day. Right now, separating the clusters is best because
>>>>> tuning a single cluster for both use cases would be difficult.
>>>>>
>>>>>
>>>>>
>>>>> > Thanks,
>>>>> > Alex
>>>>> >
>>>>> >
>>>>> > On Thu, Jun 26, 2014 at 9:28 PM, Hector Castro < [hidden email] >
>>>>> > wrote:
>>>>> >>
>>>>> >> Hi Alex,
>>>>> >>
>>>>> >> For this type of problem, you may want to look into Riak CS. [0] It
>>>>> >> is
>>>>> >> an object storage platform built on top of Riak.
>>>>> >>
>>>>> >> When you GET an object from Riak CS, you can specify a Range header
>>>>> >> to
>>>>> >> get the object by its byte offset. [1]
>>>>> >>
>>>>> >> --
>>>>> >> Hector
>>>>> >>
>>>>> >> [0] http://docs.basho.com/riakcs/latest/
>>>>> >> [1]
>>>>> >>
>>>>> >> http://docs.basho.com/riakcs/latest/references/apis/storage/s3/RiakCS-GET-Object/#Examples
>>>>> >>
>>>>> >> On Thu, Jun 26, 2014 at 3:22 PM, Alex De la rosa
>>>>> >> < [hidden email] > wrote:
>>>>> >> > Hi there,
>>>>> >> >
>>>>> >> > Imagine that I want to use Riak for a video service like Youtube
>>>>> >> > and I
>>>>> >> > save
>>>>> >> > the video file in Riak... is there a way to stream the contents of
>>>>> >> > 1
>>>>> >> > key? I
>>>>> >> > know we can stream keys... but what about its content?
>>>>> >> >
>>>>> >> > It would be pretty troublesome to have to wait for the full data
>>>>> >> > to be
>>>>> >> > downloaded $bucket->get('myvideo') to be able to serve it... as
>>>>> >> > they can
>>>>> >> > be
>>>>> >> > pretty big files.
>>>>> >> >
>>>>> >> > Is there any recommendation to store/get big files like videos? or
>>>>> >> > it
>>>>> >> > would
>>>>> >> > be better to use some other system than Riak for the job?
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> > Alex
>>>>> >> >
>>>>> >> > _______________________________________________
>>>>> >> > riak-users mailing list
>>>>> >> > [hidden email]
>>>>> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>> >> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [hidden email]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [hidden email]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>



--
Kota UENISHI / @kuenishi
Basho Japan KK


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com