Riak and PGM

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak and PGM

Neville Burnell
Does/can Riak use PGM [1] for replicating writes ?

It seems PGM would be more efficient when the number of physical nodes starts to get high.

Just curious!


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak and PGM

Neville Burnell
A quick follow up - it seems RabbitMQ now supports 0MQ [1][2] which might be a good fit with Riak

From the blog post [1]:
"0MQ is bundled with OpenPGM library which implements a reliable mutlicast protocol called PGM. The r0mq bridge thus allows to multicast messages from RabbitMQ broker to the clients (0MQ clients to be precise — AMQP has no multicast support). This kind of functionality is extremely useful in scenarios where a lot of identical data is passed to many boxes on the LAN. If a separate copy of each datum is sent to each subscriber, you can easily exceed capacity of your network. With multicast, data is sent once only to all the subscribers thus keeping the bandwidth usage constant even when the number of subscribers grows."


On 5 December 2010 17:32, Neville Burnell <[hidden email]> wrote:
Does/can Riak use PGM [1] for replicating writes ?

It seems PGM would be more efficient when the number of physical nodes starts to get high.

Just curious!



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak and PGM

Neville Burnell
And 0MQ has Erlang bindings anyhow, so rabbitMQ now required!

http://www.zeromq.org/bindings:erlang

On 5 December 2010 18:04, Neville Burnell <[hidden email]> wrote:
A quick follow up - it seems RabbitMQ now supports 0MQ [1][2] which might be a good fit with Riak

From the blog post [1]:
"0MQ is bundled with OpenPGM library which implements a reliable mutlicast protocol called PGM. The r0mq bridge thus allows to multicast messages from RabbitMQ broker to the clients (0MQ clients to be precise — AMQP has no multicast support). This kind of functionality is extremely useful in scenarios where a lot of identical data is passed to many boxes on the LAN. If a separate copy of each datum is sent to each subscriber, you can easily exceed capacity of your network. With multicast, data is sent once only to all the subscribers thus keeping the bandwidth usage constant even when the number of subscribers grows."


On 5 December 2010 17:32, Neville Burnell <[hidden email]> wrote:
Does/can Riak use PGM [1] for replicating writes ?

It seems PGM would be more efficient when the number of physical nodes starts to get high.

Just curious!




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak and PGM

Neville Burnell
that should be "RabbitMQ *not* required"

On 5 December 2010 19:27, Neville Burnell <[hidden email]> wrote:
And 0MQ has Erlang bindings anyhow, so rabbitMQ now required!

http://www.zeromq.org/bindings:erlang


On 5 December 2010 18:04, Neville Burnell <[hidden email]> wrote:
A quick follow up - it seems RabbitMQ now supports 0MQ [1][2] which might be a good fit with Riak

From the blog post [1]:
"0MQ is bundled with OpenPGM library which implements a reliable mutlicast protocol called PGM. The r0mq bridge thus allows to multicast messages from RabbitMQ broker to the clients (0MQ clients to be precise — AMQP has no multicast support). This kind of functionality is extremely useful in scenarios where a lot of identical data is passed to many boxes on the LAN. If a separate copy of each datum is sent to each subscriber, you can easily exceed capacity of your network. With multicast, data is sent once only to all the subscribers thus keeping the bandwidth usage constant even when the number of subscribers grows."


On 5 December 2010 17:32, Neville Burnell <[hidden email]> wrote:
Does/can Riak use PGM [1] for replicating writes ?

It seems PGM would be more efficient when the number of physical nodes starts to get high.

Just curious!





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak and PGM

Bob Ippolito
In reply to this post by Neville Burnell
We've looked at these recently and they're not exactly prime time.
They don't even compile on Mac OS X without a lot of prodding, at
least with our configuration. I think they also require some
development version of 0mq. That said, would be more than happy if
someone fixes these problems and builds something cool with it.

I don't think there's really any communication in the Riak model that
would benefit from better multicast. Maybe some ring stuff, but not
the reads or writes.

On Sun, Dec 5, 2010 at 3:27 PM, Neville Burnell
<[hidden email]> wrote:

> And 0MQ has Erlang bindings anyhow, so rabbitMQ now required!
> http://www.zeromq.org/bindings:erlang
>
> On 5 December 2010 18:04, Neville Burnell <[hidden email]> wrote:
>>
>> A quick follow up - it seems RabbitMQ now supports 0MQ [1][2] which might
>> be a good fit with Riak
>> From the blog post [1]:
>> "0MQ is bundled with OpenPGM library which implements a reliable mutlicast
>> protocol called PGM. The r0mq bridge thus allows to multicast messages from
>> RabbitMQ broker to the clients (0MQ clients to be precise — AMQP has no
>> multicast support). This kind of functionality is extremely useful in
>> scenarios where a lot of identical data is passed to many boxes on the LAN.
>> If a separate copy of each datum is sent to each subscriber, you can easily
>> exceed capacity of your network. With multicast, data is sent once only to
>> all the subscribers thus keeping the bandwidth usage constant even when the
>> number of subscribers grows."
>> [1] http://www.rabbitmq.com/blog/2010/10/18/rabbitmq0mq-bridge/
>> [2] https://github.com/rabbitmq/rmq-0mq
>>
>> On 5 December 2010 17:32, Neville Burnell <[hidden email]>
>> wrote:
>>>
>>> Does/can Riak use PGM [1] for replicating writes ?
>>> It seems PGM would be more efficient when the number of physical nodes
>>> starts to get high.
>>> Just curious!
>>> [1] http://en.wikipedia.org/wiki/Pragmatic_General_Multicast
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak and PGM

Neville Burnell
Hi Bob,

Thanks for your reply.

> We've looked at these recently and they're not exactly prime time.

Yes, seems that way, although the jury has mixed assessments. Ryan rejected 0MQ for NodeJS, while Zed Shaw used 0MQ to build Mongrel2.

Kind Regards

Neville

On 5 December 2010 19:49, Bob Ippolito <[hidden email]> wrote:
We've looked at these recently and they're not exactly prime time.
They don't even compile on Mac OS X without a lot of prodding, at
least with our configuration. I think they also require some
development version of 0mq. That said, would be more than happy if
someone fixes these problems and builds something cool with it.

I don't think there's really any communication in the Riak model that
would benefit from better multicast. Maybe some ring stuff, but not
the reads or writes.

On Sun, Dec 5, 2010 at 3:27 PM, Neville Burnell
<[hidden email]> wrote:
> And 0MQ has Erlang bindings anyhow, so rabbitMQ now required!
> http://www.zeromq.org/bindings:erlang
>
> On 5 December 2010 18:04, Neville Burnell <[hidden email]> wrote:
>>
>> A quick follow up - it seems RabbitMQ now supports 0MQ [1][2] which might
>> be a good fit with Riak
>> From the blog post [1]:
>> "0MQ is bundled with OpenPGM library which implements a reliable mutlicast
>> protocol called PGM. The r0mq bridge thus allows to multicast messages from
>> RabbitMQ broker to the clients (0MQ clients to be precise — AMQP has no
>> multicast support). This kind of functionality is extremely useful in
>> scenarios where a lot of identical data is passed to many boxes on the LAN.
>> If a separate copy of each datum is sent to each subscriber, you can easily
>> exceed capacity of your network. With multicast, data is sent once only to
>> all the subscribers thus keeping the bandwidth usage constant even when the
>> number of subscribers grows."
>> [1] http://www.rabbitmq.com/blog/2010/10/18/rabbitmq0mq-bridge/
>> [2] https://github.com/rabbitmq/rmq-0mq
>>
>> On 5 December 2010 17:32, Neville Burnell <[hidden email]>
>> wrote:
>>>
>>> Does/can Riak use PGM [1] for replicating writes ?
>>> It seems PGM would be more efficient when the number of physical nodes
>>> starts to get high.
>>> Just curious!
>>> [1] http://en.wikipedia.org/wiki/Pragmatic_General_Multicast
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com