API for Creation of Bucket Types

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

API for Creation of Bucket Types

superstructor
Hi all,

I am running Riak 2.0.1 in a Docker container on each physical host in the cluster.

I need to programatically create bucket types from an application running in a separate Docker container.

This was easy before using Docker, as I would just execute the riak-admin tool from my application.

With Docker there is now no direct shell access possible between the Riak container and the application container, so I cannot run riak-admin.

Each container in Docker has a virtual ethernet interface with its own IP address, so obviously I can still access the HTTP and Protobuf APIs remotely from the application container.

Is there any API available, or on the roadmap, to create bucket types ?

If not how would you suggest I create bucket types in this situation ?

Thanks,

Isaac

Reply | Threaded
Open this post in threaded view
|

Re: API for Creation of Bucket Types

Luke Bakken
Hi Isaac,

Is the lack of shell access due to how you've configured Docker? I'm
not familiar with it, however it does appear that access is possible:

https://docs.docker.com/articles/basics/#running-an-interactive-shell

Providing a bucket type API is planned for a future Riak 2.X release.

--
Luke Bakken
Engineer / CSE
[hidden email]


On Tue, Oct 14, 2014 at 12:41 AM, superstructor
<[hidden email]> wrote:

> Hi all,
>
> I am running Riak 2.0.1 in a Docker container on each physical host in the
> cluster.
>
> I need to programatically create bucket types from an application running in
> a separate Docker container.
>
> This was easy before using Docker, as I would just execute the riak-admin
> tool from my application.
>
> With Docker there is now no direct shell access possible between the Riak
> container and the application container, so I cannot run riak-admin.
>
> Each container in Docker has a virtual ethernet interface with its own IP
> address, so obviously I can still access the HTTP and Protobuf APIs remotely
> from the application container.
>
> Is there any API available, or on the roadmap, to create bucket types ?
>
> If not how would you suggest I create bucket types in this situation ?
>
> Thanks,
>
> Isaac
>
>
>
>
>
> --
> View this message in context: http://riak-users.197444.n3.nabble.com/API-for-Creation-of-Bucket-Types-tp4031932.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: API for Creation of Bucket Types

Jon Meredith
Hi Isaac,

Adding an API for bucket types is high on our list of priorities, but we do not have a timeline for availability at the moment.

I'm no Docker expert either, but I wondered if you could wrap a version of riak-admin with nsenter until we deliver it?


On Tue, Oct 14, 2014 at 8:55 AM, Luke Bakken <[hidden email]> wrote:
Hi Isaac,

Is the lack of shell access due to how you've configured Docker? I'm
not familiar with it, however it does appear that access is possible:

https://docs.docker.com/articles/basics/#running-an-interactive-shell

Providing a bucket type API is planned for a future Riak 2.X release.

--
Luke Bakken
Engineer / CSE
[hidden email]


On Tue, Oct 14, 2014 at 12:41 AM, superstructor
<[hidden email]> wrote:
> Hi all,
>
> I am running Riak 2.0.1 in a Docker container on each physical host in the
> cluster.
>
> I need to programatically create bucket types from an application running in
> a separate Docker container.
>
> This was easy before using Docker, as I would just execute the riak-admin
> tool from my application.
>
> With Docker there is now no direct shell access possible between the Riak
> container and the application container, so I cannot run riak-admin.
>
> Each container in Docker has a virtual ethernet interface with its own IP
> address, so obviously I can still access the HTTP and Protobuf APIs remotely
> from the application container.
>
> Is there any API available, or on the roadmap, to create bucket types ?
>
> If not how would you suggest I create bucket types in this situation ?
>
> Thanks,
>
> Isaac
>
>
>
>
>
> --
> View this message in context: http://riak-users.197444.n3.nabble.com/API-for-Creation-of-Bucket-Types-tp4031932.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



--
Jon Meredith
Chief Architect
Basho Technologies, Inc.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: API for Creation of Bucket Types

john.kline
In reply to this post by Luke Bakken
Luke,

Docker interactive shells are only used when running a docker image locally for testing. When running in production Docker images typically do not run SSH server, making ssh connections to the running image impossible.

Isaac,
What you may consider doing is looking at using nsenter. This utility allows you to switch to the containers namespace and execute commands from within the container without forcing you to run ssh server and and init process just to try and do riak-admin.


Cheers,
John


John Kline
P:314-495-3391 | E: [hidden email] | sh: :(){ :|:& };:


On Tue, Oct 14, 2014 at 9:55 AM, Luke Bakken <[hidden email]> wrote:
Hi Isaac,

Is the lack of shell access due to how you've configured Docker? I'm
not familiar with it, however it does appear that access is possible:

https://docs.docker.com/articles/basics/#running-an-interactive-shell

Providing a bucket type API is planned for a future Riak 2.X release.

--
Luke Bakken
Engineer / CSE
[hidden email]


On Tue, Oct 14, 2014 at 12:41 AM, superstructor
<[hidden email]> wrote:
> Hi all,
>
> I am running Riak 2.0.1 in a Docker container on each physical host in the
> cluster.
>
> I need to programatically create bucket types from an application running in
> a separate Docker container.
>
> This was easy before using Docker, as I would just execute the riak-admin
> tool from my application.
>
> With Docker there is now no direct shell access possible between the Riak
> container and the application container, so I cannot run riak-admin.
>
> Each container in Docker has a virtual ethernet interface with its own IP
> address, so obviously I can still access the HTTP and Protobuf APIs remotely
> from the application container.
>
> Is there any API available, or on the roadmap, to create bucket types ?
>
> If not how would you suggest I create bucket types in this situation ?
>
> Thanks,
>
> Isaac
>
>
>
>
>
> --
> View this message in context: http://riak-users.197444.n3.nabble.com/API-for-Creation-of-Bucket-Types-tp4031932.html
> Sent from the Riak Users mailing list archive at Nabble.com.
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: API for Creation of Bucket Types

superstructor
Thanks everyone for your replies.

I have used nsenter for interactive debugging from the host, but I havn't tried to use it for intra-container shell access. I will look into it albeit not an ideal solution.

Just to explore possibilities further, how does riak-admin actually communicate with the Riak daemon process ?

Is it possible that I could install riak-admin in both containers, and share a file or network resource between the containers that riak-admin could use to connect to the Riak daemon from the application container ? What is it exactly that riak-admin needs to connect to Riak ?

Thanks,

Isaac
Reply | Threaded
Open this post in threaded view
|

Strong consistent buckets ready after starting a cluster.

zsolt.laky
In reply to this post by superstructor
Hi All!

I am using a dev cluster with three nodes on a Mac OS. After a boot, I start
the nodes and they are ok immediately for the default buckets, but for the
strong consistent type it took 10 minutes to get ready and perform as
expected.

Here is a small example where a get returns with different values and at the
end it is ok and I can be sure the RIAK cluster is up and running for strong
consistent buckets. At the very beginning of startup I got
{error,<<"insufficient_vnodes_available">>} replies but it is ok.

My request for help is to find a way when I know the cluster is ready. I
understand, RIAK needs time to sync all the nodes after startup, on the
other hand I need to know when it gets ready for production and will give
consistent answers. Is there a command in erlang to check it? I also attach
part of the full log when I listed the keys and values in the buckets to see
when would I get the correct result. It took about 10 minutes to get there.

Thanks in advance for thoughts and suggestions, your help is highly
appreciated.
Kind regards,
Zsolt

==============================
Type:"strong"
------------------------
Bucket:<<"Actors_msuser">>
Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"timeout">>}
------------------------
Bucket:<<"Actors_mssession">>
Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"timeout">>}
ok
==============================
Type:"strong"
------------------------
Bucket:<<"Actors_msuser">>
Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"failed">>}
------------------------
Bucket:<<"Actors_mssession">>
Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"timeout">>}
==============================
Type:"strong"
------------------------
Bucket:<<"Actors_msuser">>
Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"failed">>}
------------------------
Bucket:<<"Actors_mssession">>
Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"failed">>}
==============================
Type:"strong"
------------------------
Bucket:<<"Actors_msuser">>
...........................
Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,notfound}
------------------------
Bucket:<<"Actors_mssession">>
Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,notfound}


---
This email has been checked for viruses by Avast antivirus software.
http://www.avast.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

partlog.zip (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Strong consistent buckets ready after starting a cluster.

Sargun Dhillon
In reply to this post by superstructor
I really don't recommend doing this. Really, the cluster shouldn't be
changing often enough where the ensemble status is changing all that
often. You can do this by exploiting internal APIs that are accessible
over distributed Erlang in riak_ensemble. Though, connecting to the
cluster over distributed Erlang from your client is probably also less
than recommended because it adds extra complexity, and you're poking
at internal APIs that aren't meant to be exposed to the client.

If you want to know for your own sake, you can look at the output from
'riak-admin ensemble-status'

In order to do this programmatically, you'd want to hook into
riak_ensemble_manager, and interrogate it for the status of the
ensembles. I would start by looking at the code in
riak_kv/riak_kv_ensemble_console.erl to go down this route.

On Mon, Feb 2, 2015 at 3:35 AM, Zsolt Laky <[hidden email]> wrote:

> Hi All!
>
> I am using a dev cluster with three nodes on a Mac OS. After a boot, I start
> the nodes and they are ok immediately for the default buckets, but for the
> strong consistent type it took 10 minutes to get ready and perform as
> expected.
>
> Here is a small example where a get returns with different values and at the
> end it is ok and I can be sure the RIAK cluster is up and running for strong
> consistent buckets. At the very beginning of startup I got
> {error,<<"insufficient_vnodes_available">>} replies but it is ok.
>
> My request for help is to find a way when I know the cluster is ready. I
> understand, RIAK needs time to sync all the nodes after startup, on the
> other hand I need to know when it gets ready for production and will give
> consistent answers. Is there a command in erlang to check it? I also attach
> part of the full log when I listed the keys and values in the buckets to see
> when would I get the correct result. It took about 10 minutes to get there.
>
> Thanks in advance for thoughts and suggestions, your help is highly
> appreciated.
> Kind regards,
> Zsolt
>
> ==============================
> Type:"strong"
> ------------------------
> Bucket:<<"Actors_msuser">>
> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"timeout">>}
> ------------------------
> Bucket:<<"Actors_mssession">>
> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"timeout">>}
> ok
> ==============================
> Type:"strong"
> ------------------------
> Bucket:<<"Actors_msuser">>
> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"failed">>}
> ------------------------
> Bucket:<<"Actors_mssession">>
> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"timeout">>}
> ==============================
> Type:"strong"
> ------------------------
> Bucket:<<"Actors_msuser">>
> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"failed">>}
> ------------------------
> Bucket:<<"Actors_mssession">>
> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"failed">>}
> ==============================
> Type:"strong"
> ------------------------
> Bucket:<<"Actors_msuser">>
> ...........................
> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,notfound}
> ------------------------
> Bucket:<<"Actors_mssession">>
> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,notfound}
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> http://www.avast.com
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Strong consistent buckets ready after starting a cluster.

zsolt.laky
Hi Dhillon,

Thanks for the answer. You sched the light.

As I need it only during dev, riak-admin ensemble-status will help I believe.

cheers,
zsolt

> On 06 Feb 2015, at 21:41, Sargun Dhillon <[hidden email]> wrote:
>
> I really don't recommend doing this. Really, the cluster shouldn't be
> changing often enough where the ensemble status is changing all that
> often. You can do this by exploiting internal APIs that are accessible
> over distributed Erlang in riak_ensemble. Though, connecting to the
> cluster over distributed Erlang from your client is probably also less
> than recommended because it adds extra complexity, and you're poking
> at internal APIs that aren't meant to be exposed to the client.
>
> If you want to know for your own sake, you can look at the output from
> 'riak-admin ensemble-status'
>
> In order to do this programmatically, you'd want to hook into
> riak_ensemble_manager, and interrogate it for the status of the
> ensembles. I would start by looking at the code in
> riak_kv/riak_kv_ensemble_console.erl to go down this route.
>
>> On Mon, Feb 2, 2015 at 3:35 AM, Zsolt Laky <[hidden email]> wrote:
>> Hi All!
>>
>> I am using a dev cluster with three nodes on a Mac OS. After a boot, I start
>> the nodes and they are ok immediately for the default buckets, but for the
>> strong consistent type it took 10 minutes to get ready and perform as
>> expected.
>>
>> Here is a small example where a get returns with different values and at the
>> end it is ok and I can be sure the RIAK cluster is up and running for strong
>> consistent buckets. At the very beginning of startup I got
>> {error,<<"insufficient_vnodes_available">>} replies but it is ok.
>>
>> My request for help is to find a way when I know the cluster is ready. I
>> understand, RIAK needs time to sync all the nodes after startup, on the
>> other hand I need to know when it gets ready for production and will give
>> consistent answers. Is there a command in erlang to check it? I also attach
>> part of the full log when I listed the keys and values in the buckets to see
>> when would I get the correct result. It took about 10 minutes to get there.
>>
>> Thanks in advance for thoughts and suggestions, your help is highly
>> appreciated.
>> Kind regards,
>> Zsolt
>>
>> ==============================
>> Type:"strong"
>> ------------------------
>> Bucket:<<"Actors_msuser">>
>> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"timeout">>}
>> ------------------------
>> Bucket:<<"Actors_mssession">>
>> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"timeout">>}
>> ok
>> ==============================
>> Type:"strong"
>> ------------------------
>> Bucket:<<"Actors_msuser">>
>> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"failed">>}
>> ------------------------
>> Bucket:<<"Actors_mssession">>
>> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"timeout">>}
>> ==============================
>> Type:"strong"
>> ------------------------
>> Bucket:<<"Actors_msuser">>
>> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,<<"failed">>}
>> ------------------------
>> Bucket:<<"Actors_mssession">>
>> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,<<"failed">>}
>> ==============================
>> Type:"strong"
>> ------------------------
>> Bucket:<<"Actors_msuser">>
>> ...........................
>> Key:"{msuser_actor,\"z2\",lamardan}" Value:Error:{error,notfound}
>> ------------------------
>> Bucket:<<"Actors_mssession">>
>> Key:"{mssession_actor,\"z2\",zschat,0}" Value:Error:{error,notfound}
>>
>>
>> ---
>> This email has been checked for viruses by Avast antivirus software.
>> http://www.avast.com
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com