Copying data between clusters

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Copying data between clusters

Alan McConnell-2
  • Is it possible to use the built-in backup/restore functions of riak-admin to copy data between clusters?  The scenario is that we have separate test and production environments, each with separate riak clusters of different sizes and configurations.  Every so often we'd like to copy the data from production into test.  Can I run 'backup all' in prod, then run restore on the resultant file in test, and have everything work?  Or does restore only work on the same cluster configuration that the backup was performed on?  What's the easiest way of doing this?
  • I've noticed that backup takes a very long time, and results in a file that is much larger than the actual cluster data directories of the cluster.  Is this normal?
  • Not related to backups, but how do I configure the global replication count of the cluster?

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Copying data between clusters

Kevin Smith-5
Alan -

I can't answer your replication questions -- don't have a lot of experience with it yet -- but I can take a stab at setting the global replication setting.

If I'm understanding correctly I believe you're referring to the N value. N controls how many replicas of an entry Riak stores, in total, around the cluster. Docs on tuning this value can be found here: http://wiki.basho.com/display/RIAK/Replication

--Kevin


On May 7, 2010, at 10:06 PM, Alan McConnell wrote:

> • Is it possible to use the built-in backup/restore functions of riak-admin to copy data between clusters?  The scenario is that we have separate test and production environments, each with separate riak clusters of different sizes and configurations.  Every so often we'd like to copy the data from production into test.  Can I run 'backup all' in prod, then run restore on the resultant file in test, and have everything work?  Or does restore only work on the same cluster configuration that the backup was performed on?  What's the easiest way of doing this?
> • I've noticed that backup takes a very long time, and results in a file that is much larger than the actual cluster data directories of the cluster.  Is this normal?
> • Not related to backups, but how do I configure the global replication count of the cluster?
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Copying data between clusters

Alan McConnell-2
Thanks Kevin,
Is there a way to set a default N (and R/W) value across a cluster via configuration?  I know there's a default_props feature in configuration but I don't know if it's possible to use that for default N/R/W values and I don't know what the syntax is.

In case anyone was wondering I went ahead and tried a backup/restore between production and test and it worked just fine.  Restoring from a backup file will work across different clusters.

On Fri, May 7, 2010 at 9:17 PM, Kevin Smith <[hidden email]> wrote:
Alan -

I can't answer your replication questions -- don't have a lot of experience with it yet -- but I can take a stab at setting the global replication setting.

If I'm understanding correctly I believe you're referring to the N value. N controls how many replicas of an entry Riak stores, in total, around the cluster. Docs on tuning this value can be found here: http://wiki.basho.com/display/RIAK/Replication

--Kevin


On May 7, 2010, at 10:06 PM, Alan McConnell wrote:

>       • Is it possible to use the built-in backup/restore functions of riak-admin to copy data between clusters?  The scenario is that we have separate test and production environments, each with separate riak clusters of different sizes and configurations.  Every so often we'd like to copy the data from production into test.  Can I run 'backup all' in prod, then run restore on the resultant file in test, and have everything work?  Or does restore only work on the same cluster configuration that the backup was performed on?  What's the easiest way of doing this?
>       • I've noticed that backup takes a very long time, and results in a file that is much larger than the actual cluster data directories of the cluster.  Is this normal?
>       • Not related to backups, but how do I configure the global replication count of the cluster?
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Copying data between clusters

Mårten Gustafson
N-val is configurable through default bucket props.

AFAIK there's currently no way change server defaults for R, W and DW.



/Mårten

On 8 maj 2010, at 20.27, Alan McConnell <[hidden email]> wrote:

Thanks Kevin,
Is there a way to set a default N (and R/W) value across a cluster via configuration?  I know there's a default_props feature in configuration but I don't know if it's possible to use that for default N/R/W values and I don't know what the syntax is.

In case anyone was wondering I went ahead and tried a backup/restore between production and test and it worked just fine.  Restoring from a backup file will work across different clusters.

On Fri, May 7, 2010 at 9:17 PM, Kevin Smith <[hidden email]> wrote:
Alan -

I can't answer your replication questions -- don't have a lot of experience with it yet -- but I can take a stab at setting the global replication setting.

If I'm understanding correctly I believe you're referring to the N value. N controls how many replicas of an entry Riak stores, in total, around the cluster. Docs on tuning this value can be found here: http://wiki.basho.com/display/RIAK/Replication

--Kevin


On May 7, 2010, at 10:06 PM, Alan McConnell wrote:

>       • Is it possible to use the built-in backup/restore functions of riak-admin to copy data between clusters?  The scenario is that we have separate test and production environments, each with separate riak clusters of different sizes and configurations.  Every so often we'd like to copy the data from production into test.  Can I run 'backup all' in prod, then run restore on the resultant file in test, and have everything work?  Or does restore only work on the same cluster configuration that the backup was performed on?  What's the easiest way of doing this?
>       • I've noticed that backup takes a very long time, and results in a file that is much larger than the actual cluster data directories of the cluster.  Is this normal?
>       • Not related to backups, but how do I configure the global replication count of the cluster?
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com