Help with local restore for dev enviroment

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with local restore for dev enviroment

Mark Wagner
Hey all,

I'm new to riak and I'm working on an ETL script that needs to pull data from a riak cluster.

My client has sent me a backup from one of their cluster nodes. bitcask data,. rings and config.

Here is what I am doing:   on OSX

# Install riak on my local OSX development box 
brew install riak

#goto the default data directory setup by brew
cd ~/Developer/Cellar/riak/1.3.1-x86_64/libexec/data/

# create a bitcask directory and copy the bitcask data into it
mkdir bitcask
cd bitcask 
cp -r ~/Downloads/riak/var/lib/riak/bitcask/* ./

#goto the ring directory and copy the ring data into it
cd ../ring
cp -r ~/Downloads/riak/var/lib/riak/ring/* ./

#goto the etc directory, delete pem files, copy over the pem files from backup
cd ../../etc/
rm *.pem
cp ~/Downloads/riak/etc/riak/*.pem ./

I do not believe I need to make any changes to the default vm.args or app.config files installed by brew.

#run the riak-admin reip command since the hostname is different

At this point I believe I should be able to start riak and browse the data.   Is this correct?
How can I verify, that riak see's the data?
I have no idea of the structure of the data how can I determine the structure?

Thanks for your help,

Mark






--
Mark Wagner, Web & Mobile Developer
:coderow
Web & mobile application development for business
www.coderow.com | mark@... | @thesociableme




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Help with local restore for dev enviroment

Shane McEwan-2
On 09/07/13 22:24, Mark Wagner wrote:

> Hey all,
>
> I'm new to riak and I'm working on an ETL script that needs to pull data
> from a riak cluster.
>
> My client has sent me a backup from one of their cluster nodes. bitcask
> data,. rings and config.
>
> *snip*
>
> At this point I believe I should be able to start riak and browse the data.
>    Is this correct?

My suggestion would be to set up Riak and get it running before copying
over any data. If you're running a cluster get your nodes joined
together and communicating on the ring. Once you're happy that your Riak
node(s) are running, shut them down and copy _just_ the bitcask data
directory from the backup into the correct location for your node(s).

Don't mess with the ring files and don't worry about "reip".

Start up Riak.

Riak is smart enough to find the data you've copied in. If you're
running a cluster then it's important to copy the correct node's data
onto the new nodes. It's not straightforward to know which old node
corresponds to which new node but Riak will cope if you get it wrong. It
will just hand off partitions to the correct nodes. A bit of trial and
error allows you to figure out which nodes are equivalent between setups
to avoid unnecessary handoffs.

(Note: I've only done this with the LevelDB backend but I would expect
Bitcask to work in much the same way.)

Hope this helps!

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Help with local restore for dev enviroment

Justin Sheehy
In reply to this post by Mark Wagner
Hi, Mark.

You've already received a little advice generally so I won't pile on that part, but one thing stood out to me:

> My client has sent me a backup from one of their cluster nodes. bitcask data,. rings and config.

Unless I'm misunderstanding what you're doing, what you're working on will not get you the data from the whole cluster, but the fraction of the data that was stored on the one node that you have a backup from. Just a warning, in case you hadn't realized this.

-Justin



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Help with local restore for dev enviroment

Mark Wagner
Thanks for the info!!!  I appreciate the help and I am glad to know that it should just work!

I don't really need the data from the whole cluster while developing the script.  I just need to get my queries working etc...

One issue I am facing is I don't know the structure of the data.  So I am trying to browse it (http://localhost:8098/riak?buckets=true)  and I getting content decode  error.

Any ideas?  thoughts on how to troubleshoot this?  Or different approaches to determining the structure?


Thanks AGAIN!!!


Mark


On Wed, Jul 10, 2013 at 7:59 AM, Justin Sheehy <[hidden email]> wrote:
Hi, Mark.

You've already received a little advice generally so I won't pile on that part, but one thing stood out to me:

> My client has sent me a backup from one of their cluster nodes. bitcask data,. rings and config.

Unless I'm misunderstanding what you're doing, what you're working on will not get you the data from the whole cluster, but the fraction of the data that was stored on the one node that you have a backup from. Just a warning, in case you hadn't realized this.

-Justin





--
Mark Wagner, Web & Mobile Developer
:coderow
Web & mobile application development for business
www.coderow.com | [hidden email] | @thesociableme




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Help with local restore for dev enviroment

Mark Phillips-4
Hi Mark,

If you need to get the structure the data, your best bet is to pull the content type. You'll need to list-keys [1] on that bucket and then parse the header for each key with a HEAD request, paying special attention to the content-type for the key [2]. Keep in mind we don't recommend using list-keys for production, but if it's for test data and an offline-cluster, it's fine. Make sure to use the "?keys=streamoption if the client you're using supports it. 

The biggest caveat here is that you're trusting the client that stored the data initially to have specified the correct content type. Riak doesn't  validate it proactively on the way in. 

Hope that helps. 

Mark 




On Wed, Jul 10, 2013 at 7:37 AM, Mark Wagner <[hidden email]> wrote:
Thanks for the info!!!  I appreciate the help and I am glad to know that it should just work!

I don't really need the data from the whole cluster while developing the script.  I just need to get my queries working etc...

One issue I am facing is I don't know the structure of the data.  So I am trying to browse it (http://localhost:8098/riak?buckets=true)  and I getting content decode  error.

Any ideas?  thoughts on how to troubleshoot this?  Or different approaches to determining the structure?


Thanks AGAIN!!!


Mark


On Wed, Jul 10, 2013 at 7:59 AM, Justin Sheehy <[hidden email]> wrote:
Hi, Mark.

You've already received a little advice generally so I won't pile on that part, but one thing stood out to me:

> My client has sent me a backup from one of their cluster nodes. bitcask data,. rings and config.

Unless I'm misunderstanding what you're doing, what you're working on will not get you the data from the whole cluster, but the fraction of the data that was stored on the one node that you have a backup from. Just a warning, in case you hadn't realized this.

-Justin





--
Mark Wagner, Web & Mobile Developer
:coderow
Web & mobile application development for business
www.coderow.com | [hidden email] | @thesociableme




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com