Luwak EOL

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Luwak EOL

Dave Smith
Hi all,
I wanted to take a moment and let the community-at-large know that we
are going to end-of-life the Luwak functionality in Riak, as part of
our 1.1 release in February. This simply means that the Luwak repo on
Github will not be actively developed/supported by Basho, or included
as a default Riak dependency. In keeping with our commitment to Open
Source, the code will still be available if someone else wants to
develop on it. You can also continue to use it with Riak if you are
willing to edit the deps and compile from source -- the Luwak README
will contain information on how to do this.
While the idea of Luwak was interesting, we ultimately decided that it
wasn’t the architectural path we wanted to pursue for storing larger
values in Riak. If Luwak is a piece of your Riak deployment, we’d love
to hear from you and incorporate your feedback into future directions.
Please don’t hesitate to reach out to myself or Mark Phillips.
Thanks,
D.
--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

francisco treacy-2
I can't say I'm thrilled to hear this. We use Luwak extensively.

> While the idea of Luwak was interesting, we ultimately decided that it
> wasn’t the architectural path we wanted to pursue for storing larger
> values in Riak.

Does this mean there is still an intention to support storing larger
values in Riak in the future?

Thanks,
Francisco


2011/12/19 David Smith <[hidden email]>:

> Hi all,
> I wanted to take a moment and let the community-at-large know that we
> are going to end-of-life the Luwak functionality in Riak, as part of
> our 1.1 release in February. This simply means that the Luwak repo on
> Github will not be actively developed/supported by Basho, or included
> as a default Riak dependency. In keeping with our commitment to Open
> Source, the code will still be available if someone else wants to
> develop on it. You can also continue to use it with Riak if you are
> willing to edit the deps and compile from source -- the Luwak README
> will contain information on how to do this.
> While the idea of Luwak was interesting, we ultimately decided that it
> wasn’t the architectural path we wanted to pursue for storing larger
> values in Riak. If Luwak is a piece of your Riak deployment, we’d love
> to hear from you and incorporate your feedback into future directions.
> Please don’t hesitate to reach out to myself or Mark Phillips.
> Thanks,
> D.
> --
> Dave Smith
> Director, Engineering
> Basho Technologies, Inc.
> [hidden email]
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Paul Gross
>> While the idea of Luwak was interesting, we ultimately decided that it
>> wasn’t the architectural path we wanted to pursue for storing larger
>> values in Riak.
> Does this mean there is still an intention to support storing larger
> values in Riak in the future?
Or is this something the client libraries should implement?

Thanks,
Paul
www.pgrs.net

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Dave Smith
On Tue, Dec 20, 2011 at 11:08 PM, Paul Gross <[hidden email]> wrote:

>> Does this mean there is still an intention to support storing larger
>> values in Riak in the future?
>
> Or is this something the client libraries should implement?

You could certainly implement the necessary chunking and such in
client libraries. However, it's a pretty big set of functionality and
getting it to work right with in partitioned circumstances (i.e. with
siblings) can be a challenge. As such, I don't expect that to be part
of the standard client libraries anytime soon.

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

John Axel Eriksson
Wow. Is this for real? I almost can't believe it. This is the feature we use most in Riak, we really depend on it and have quite a lot of data stored. Suddenly Riak became a dead-end for us. We've used Riak for more than a year and have invested quite a lot of time in it. This is indeed very sad news, not at all what I wanted for christmas :-(.

Does anyone have any experience with other similar replicated data stores with an http interface (along the lines of Luwak or S3 but neither of those two of course)? Any suggestions from others in our situation?

On Wed, Dec 21, 2011 at 8:16 PM, David Smith <[hidden email]> wrote:
On Tue, Dec 20, 2011 at 11:08 PM, Paul Gross <[hidden email]> wrote:

>> Does this mean there is still an intention to support storing larger
>> values in Riak in the future?
>
> Or is this something the client libraries should implement?

You could certainly implement the necessary chunking and such in
client libraries. However, it's a pretty big set of functionality and
getting it to work right with in partitioned circumstances (i.e. with
siblings) can be a challenge. As such, I don't expect that to be part
of the standard client libraries anytime soon.

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

RE: Luwak EOL

Gary Smith
In reply to this post by Dave Smith
> >> Does this mean there is still an intention to support storing larger
> >> values in Riak in the future?
> >
> > Or is this something the client libraries should implement?
>
> You could certainly implement the necessary chunking and such in client
> libraries. However, it's a pretty big set of functionality and getting it to
> work right with in partitioned circumstances (i.e. with
> siblings) can be a challenge. As such, I don't expect that to be part of the
> standard client libraries anytime soon.
>

I think this pretty much kills why I started playing with riak in the first place; local version of S3.

To me it seems that this is turning into another simple database, for which there are way too many right now.  I had wrote my own storage engine prior to finding riak which basically stored files and made sure it was replicated to x additional nodes (as well as s3) and the index was stored in a replicated mysql database.  It was my hope that this would be the replacement for that.

So, I guess I'm saying that, in my opinion, this project is going in a direction that no longer really supports my needs/goals.  I think it has potential still, but it needs to find a clear direction.




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Dmitry Rozhkov
In reply to this post by Dave Smith
Hi,

That's really sad. My project relies on LUWAK.
Is there any plan to introduce a replacement for storing large values?

BR,
Dima

2011/12/20 David Smith <[hidden email]>:

> Hi all,
> I wanted to take a moment and let the community-at-large know that we
> are going to end-of-life the Luwak functionality in Riak, as part of
> our 1.1 release in February. This simply means that the Luwak repo on
> Github will not be actively developed/supported by Basho, or included
> as a default Riak dependency. In keeping with our commitment to Open
> Source, the code will still be available if someone else wants to
> develop on it. You can also continue to use it with Riak if you are
> willing to edit the deps and compile from source -- the Luwak README
> will contain information on how to do this.
> While the idea of Luwak was interesting, we ultimately decided that it
> wasn’t the architectural path we wanted to pursue for storing larger
> values in Riak. If Luwak is a piece of your Riak deployment, we’d love
> to hear from you and incorporate your feedback into future directions.
> Please don’t hesitate to reach out to myself or Mark Phillips.
> Thanks,
> D.
> --
> Dave Smith
> Director, Engineering
> Basho Technologies, Inc.
> [hidden email]
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Witeman Zheng
Dear Basho Team,

Can you explain more detailed reason for EOL Luwak?  And is there some replacement or alternative solution for the part Luwak taking in Riak?

Best regards,
Zheng Zhibin

在 2011-12-23,下午6:51,Dmitry Rozhkov <[hidden email]> 写道:

> Hi,
>
> That's really sad. My project relies on LUWAK.
> Is there any plan to introduce a replacement for storing large values?
>
> BR,
> Dima
>
> 2011/12/20 David Smith <[hidden email]>:
>> Hi all,
>> I wanted to take a moment and let the community-at-large know that we
>> are going to end-of-life the Luwak functionality in Riak, as part of
>> our 1.1 release in February. This simply means that the Luwak repo on
>> Github will not be actively developed/supported by Basho, or included
>> as a default Riak dependency. In keeping with our commitment to Open
>> Source, the code will still be available if someone else wants to
>> develop on it. You can also continue to use it with Riak if you are
>> willing to edit the deps and compile from source -- the Luwak README
>> will contain information on how to do this.
>> While the idea of Luwak was interesting, we ultimately decided that it
>> wasn’t the architectural path we wanted to pursue for storing larger
>> values in Riak. If Luwak is a piece of your Riak deployment, we’d love
>> to hear from you and incorporate your feedback into future directions.
>> Please don’t hesitate to reach out to myself or Mark Phillips.
>> Thanks,
>> D.
>> --
>> Dave Smith
>> Director, Engineering
>> Basho Technologies, Inc.
>> [hidden email]
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Soren Hansen
In reply to this post by John Axel Eriksson
2011/12/21 John Axel Eriksson <[hidden email]>:
> Does anyone have any experience with other similar replicated data stores
> with an http interface (along the lines of Luwak or S3 but neither of those
> two of course)? Any suggestions from others in our situation?

OpenStack Storage a.k.a. Swift is what Rackspace uses to provide its
Cloud Files service, which is comparable to S3. It is in use in many
other organisations and service providers as well.

   http://swift.openstack.org/

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Soren Hansen
2012/1/2 Soren Hansen <[hidden email]>:
> OpenStack Storage a.k.a. Swift is what Rackspace uses to provide its
> Cloud Files service, which is comparable to S3. It is in use in many
> other organisations and service providers as well.
>
>   http://swift.openstack.org/

Oh, and it has an S3-compatible frontend as well. I've never used
that, though, so I can't say much about it.

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Mike Milner
I've been looking at CEPH a lot lately.  Object storage in a cluster of identical nodes.  It can be used for hosting raw block devices (like Amazon EBS) or object storage over http (like S3).  It works like luwak - objects are chunked into smaller pieces and replicated around the cluster independently.

Project:

S3 Interface:

Mike

On Mon, Jan 2, 2012 at 12:25 PM, Soren Hansen <[hidden email]> wrote:
2012/1/2 Soren Hansen <[hidden email]>:
> OpenStack Storage a.k.a. Swift is what Rackspace uses to provide its
> Cloud Files service, which is comparable to S3. It is in use in many
> other organisations and service providers as well.
>
>   http://swift.openstack.org/

Oh, and it has an S3-compatible frontend as well. I've never used
that, though, so I can't say much about it.

--
Soren Hansen        | http://linux2go.dk/
Ubuntu Developer    | http://www.ubuntu.com/
OpenStack Developer | http://www.openstack.org/

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

francisco treacy-2
A much more simplistic approach (depending on your write patterns and
amount of nodes) could make use of rsync.

It is used here combined with inotify:
http://denimboy.posterous.com/quick-bash-script-uses-inotify-and-rsync-to-m

Sounds like in the right direction for our use-case.

Francisco


2012/1/3 Mike Milner <[hidden email]>:

> I've been looking at CEPH a lot lately.  Object storage in a cluster of
> identical nodes.  It can be used for hosting raw block devices (like Amazon
> EBS) or object storage over http (like S3).  It works like luwak - objects
> are chunked into smaller pieces and replicated around the
> cluster independently.
>
> Project:
> http://ceph.newdream.net/
>
> S3 Interface:
> http://ceph.newdream.net/wiki/RADOS_Gateway
>
> Mike
>
>
> On Mon, Jan 2, 2012 at 12:25 PM, Soren Hansen <[hidden email]> wrote:
>>
>> 2012/1/2 Soren Hansen <[hidden email]>:
>> > OpenStack Storage a.k.a. Swift is what Rackspace uses to provide its
>> > Cloud Files service, which is comparable to S3. It is in use in many
>> > other organisations and service providers as well.
>> >
>> >   http://swift.openstack.org/
>>
>> Oh, and it has an S3-compatible frontend as well. I've never used
>> that, though, so I can't say much about it.
>>
>> --
>> Soren Hansen        | http://linux2go.dk/
>> Ubuntu Developer    | http://www.ubuntu.com/
>> OpenStack Developer | http://www.openstack.org/
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Mikhail Sobolev
In reply to this post by Witeman Zheng
On Fri, Dec 23, 2011 at 09:46:41PM +0800, Zheng Zhibin wrote:
> Dear Basho Team,
>
> Can you explain more detailed reason for EOL Luwak?  And is there some
> replacement or alternative solution for the part Luwak taking in Riak?
Seconding Zheng's request: would it be possible to hear the reasons for
Luwak EOL?

--
Misha

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Dave Smith
On Thu, Jan 12, 2012 at 12:40 PM, Mikhail Sobolev <[hidden email]> wrote:
>>
>> Can you explain more detailed reason for EOL Luwak?  And is there some
>> replacement or alternative solution for the part Luwak taking in Riak?
> Seconding Zheng's request: would it be possible to hear the reasons for
> Luwak EOL?

Hi Misha, Zheng,

From a technical standpoint, Luwak has some problems with the way in
which files are chunked and manifest of those chunks is maintained.
Specifically, there is no good way of knowing when a given chunk is no
longer in use, which creates a very thorny distributed garbage
collection issue -- this is a fundamental design problem. In addition,
the manifest of chunks does not properly handle situations where
siblings of the manifest occur -- this is a significant implementation
problem. These two issues, in particular, have led us to stop further
development of Luwak and investigate other architectural approaches.

Above and beyond these technical issues, Luwak also doesn't solve a
host of other problems that most people who want to store files in
Riak encounter. We are in the process of building a complete solution
for multi-tenancy file storage in Riak -- more info on that will be
available next month.

Hope that helps,

D.

--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[hidden email]

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Luwak EOL

Mikhail Sobolev
Hi Dave,

On Thu, Jan 12, 2012 at 01:53:46PM -0700, David Smith wrote:
> [a description of the problems/reasons]
>
> Hope that helps,
Yes, it does.  Thank you very much.

--
Misha

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

signature.asc (196 bytes) Download Attachment