riakkit, a python riak object mapper, has hit beta!

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

riakkit, a python riak object mapper, has hit beta!

Shuhao Wu
Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Sean Carey
Keep up the great work Shuhao!

On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Shuhao Wu
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Shuhao Wu
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao


On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
In reply to this post by Shuhao Wu
First of all, yes avoiding PEP8 style is a big deal for me. I don't like the inconsistencies it brings to my code. Sorry for GPL, it's a typo. I just don't like GPL-based licenses, including LGPL. I think it's overcomplicated.

Regardless "Track dependencies between models". It's the same thing as tracking relationships, but has more flexibility. I my particular example, you reference the messages by index. Basically, it's some sort of configurable relationship tracking.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 8:02 AM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

licenses (was Re: riakkit, a python riak object mapper, has hit beta!(

Justin Sheehy
Hi, Andrey.

On Mar 1, 2012, at 10:18 PM, "Andrey V. Martyanov" <[hidden email]> wrote:

> Sorry for GPL, it's a typo. I just don't like GPL-based licenses, including LGPL. I think it's overcomplicated.

You are of course free to dislike anything you wish, but it is worth mentioning that GPL and LGPL are very different licenses; the LGPL is missing infectious aspects of the GPL.

There are many projects which could not use GPL code compatibly with their preferred license but which can safely use LGPL code.

Justin



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
In reply to this post by Shuhao Wu
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com







_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: licenses (was Re: riakkit, a python riak object mapper, has hit beta!(

Andrey V. Martyanov
In reply to this post by Justin Sheehy
Hi Justin,

Sorry for the late response, I didn't  see your message! In fact, I know the differences between the two. But, what is the profit of using it? Why don't just use BSD, for example, like many open source projects do. The biggest minus of LGPL is that many people think that it's the same as GPL and have problems understanding it. Even your think that I don't know the difference! :) Why? Because, it's a common practice. A lot of people really don't know the difference. That's why I said before that (L)GPL is overcomplicated. If you open the LGPL main page [1], first thing you will see is "Why you shouldn't use the Lesser GPL for your next library". Is it normal? It confuses people. There are a lot of profit in pulling back the changes you've made - a lot of people see it, fix it, comment it, improve it and so on. Why the license forces me to to that? It shouldn't.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 8:29 AM, Justin Sheehy <[hidden email]> wrote:
Hi, Andrey.

On Mar 1, 2012, at 10:18 PM, "Andrey V. Martyanov" <[hidden email]> wrote:

> Sorry for GPL, it's a typo. I just don't like GPL-based licenses, including LGPL. I think it's overcomplicated.

You are of course free to dislike anything you wish, but it is worth mentioning that GPL and LGPL are very different licenses; the LGPL is missing infectious aspects of the GPL.

There are many projects which could not use GPL code compatibly with their preferred license but which can safely use LGPL code.

Justin




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
In reply to this post by Andrey V. Martyanov
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.

Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com






_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
In reply to this post by Andrey V. Martyanov
Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com






_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Jared Morrow
Andrey,

First off, you are indeed sending two emails to the Riak Users list each time you email, not just one, so I think gmail is doing the right thing the first time.

Second, I'm sure Shuhao would love some pull requests to help with Riakkit since you seem to have a lot of ideas to make it better.

-Jared


On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:

Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Shuhao Wu
Actually it's okay for no pull request for now.. I'm in the process of rethinking the whole framework as well as I'm having some of the issues that Andrey is describing as well.

Shuhao


On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <[hidden email]> wrote:
Andrey,

First off, you are indeed sending two emails to the Riak Users list each time you email, not just one, so I think gmail is doing the right thing the first time.

Second, I'm sure Shuhao would love some pull requests to help with Riakkit since you seem to have a lot of ideas to make it better.

-Jared


On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:

Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Shuhao Wu
Also, what do you mean by conflict resolution? Synchronization or conflicts between references such as deleting references to a document when that document is deleted?

Shuhao


On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <[hidden email]> wrote:
Actually it's okay for no pull request for now.. I'm in the process of rethinking the whole framework as well as I'm having some of the issues that Andrey is describing as well.

Shuhao



On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <[hidden email]> wrote:
Andrey,

First off, you are indeed sending two emails to the Riak Users list each time you email, not just one, so I think gmail is doing the right thing the first time.

Second, I'm sure Shuhao would love some pull requests to help with Riakkit since you seem to have a lot of ideas to make it better.

-Jared


On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:

Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Andrey V. Martyanov
Basically, I meant sibling reconciliation.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 8:01 PM, Shuhao Wu <[hidden email]> wrote:
Also, what do you mean by conflict resolution? Synchronization or conflicts between references such as deleting references to a document when that document is deleted?

Shuhao



On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <[hidden email]> wrote:
Actually it's okay for no pull request for now.. I'm in the process of rethinking the whole framework as well as I'm having some of the issues that Andrey is describing as well.

Shuhao



On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <[hidden email]> wrote:
Andrey,

First off, you are indeed sending two emails to the Riak Users list each time you email, not just one, so I think gmail is doing the right thing the first time.

Second, I'm sure Shuhao would love some pull requests to help with Riakkit since you seem to have a lot of ideas to make it better.

-Jared


On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:

Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: licenses (was Re: riakkit, a python riak object mapper, has hit beta!(

Shuhao Wu
In reply to this post by Andrey V. Martyanov
It's asking them to use GPL, which is even more restrictive, but a better move for general open source.

LGPL is what I feel like the best for this library, as I want a copyleft license.

Shuhao


On Fri, Mar 2, 2012 at 5:24 AM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Justin,

Sorry for the late response, I didn't  see your message! In fact, I know the differences between the two. But, what is the profit of using it? Why don't just use BSD, for example, like many open source projects do. The biggest minus of LGPL is that many people think that it's the same as GPL and have problems understanding it. Even your think that I don't know the difference! :) Why? Because, it's a common practice. A lot of people really don't know the difference. That's why I said before that (L)GPL is overcomplicated. If you open the LGPL main page [1], first thing you will see is "Why you shouldn't use the Lesser GPL for your next library". Is it normal? It confuses people. There are a lot of profit in pulling back the changes you've made - a lot of people see it, fix it, comment it, improve it and so on. Why the license forces me to to that? It shouldn't.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 8:29 AM, Justin Sheehy <[hidden email]> wrote:
Hi, Andrey.

On Mar 1, 2012, at 10:18 PM, "Andrey V. Martyanov" <[hidden email]> wrote:

> Sorry for GPL, it's a typo. I just don't like GPL-based licenses, including LGPL. I think it's overcomplicated.

You are of course free to dislike anything you wish, but it is worth mentioning that GPL and LGPL are very different licenses; the LGPL is missing infectious aspects of the GPL.

There are many projects which could not use GPL code compatibly with their preferred license but which can safely use LGPL code.

Justin




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: riakkit, a python riak object mapper, has hit beta!

Michael Clemmons
In reply to this post by Andrey V. Martyanov
Andrey / Shuhao,
I had been working on a similar project to riakkit.  Its not near as complete but had some nice feature like lazy loading and basically the same bucket/client defined class structure.
I have similar concerns as Andrey about riakkit's current beta pep8 compliance being a big one, but that could be fixed without much fuss just time.
I think having a pythonista friendly client and object mapper is of key importance and something I'd like to see handled in a standards oriented way.  If your interested in having a discussion about designs for riakkit let me know, probably better off the list though.

Shuhao,
"""
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)
"""
I tried something similar in my obj mapper, I baically subclasses the RiakBucket and bypased the RiakObject for lazy loading and better control.  This was nice at friend but became tedious and seems like itll have issues in the future since i was having to access internal methods.  If I were to do it again I think I will probably use the basho api and abstract it in the mapper.

On Fri, Mar 2, 2012 at 8:26 AM, Andrey V. Martyanov <[hidden email]> wrote:
Basically, I meant sibling reconciliation.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:01 PM, Shuhao Wu <[hidden email]> wrote:
Also, what do you mean by conflict resolution? Synchronization or conflicts between references such as deleting references to a document when that document is deleted?

Shuhao



On Fri, Mar 2, 2012 at 9:56 AM, Shuhao Wu <[hidden email]> wrote:
Actually it's okay for no pull request for now.. I'm in the process of rethinking the whole framework as well as I'm having some of the issues that Andrey is describing as well.

Shuhao



On Fri, Mar 2, 2012 at 9:51 AM, Jared Morrow <[hidden email]> wrote:
Andrey,

First off, you are indeed sending two emails to the Riak Users list each time you email, not just one, so I think gmail is doing the right thing the first time.

Second, I'm sure Shuhao would love some pull requests to help with Riakkit since you seem to have a lot of ideas to make it better.

-Jared


On Mar 2, 2012, at 6:39 AM, Andrey V. Martyanov wrote:

Also, I think we can add RoR-like migrations on top of that with automatic schema versioning and all the stuff.

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:22 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm sorry for the repost, but my gmail client by default sends an email to a specific subscriber, not to riak-users.


Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 6:17 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi Shuhao,

I don't have enough time to write my own implementation and to see how it works in practice. But I share my thoughts about that.
It's possible to save the current API, it's common now and used in SQLAlchemy and Django ORM. But underlying implementation I see a bit different. The simple workflow:
1. Define the models with specific fields and properties;
2. Generate the JSON Schema [1] based on these fields;
3. Create an instances of our models;
4. Validate against the schema;
5. Generate the optimal sequence of actions we need to do to store the data, based on the meta information of our models;
6. Store the data, resolve the conflicts if occur.

Also it should be possible to deserialize the JSON objects into python objects.


Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 5:47 PM, Shuhao Wu <[hidden email]> wrote:

Yeah. You're looking at something like dictshield. As riakkit is more a RAD tool in its current state. You bring in an excellent point, nonetheless, which gave me some intuition and ideas about the next step.

Cheers,
Shuhao
Sent from my phone.

On Mar 1, 2012 11:41 PM, "Andrey V. Martyanov" <[hidden email]> wrote:
Riakkit in its current implementation has a  lot of overhead. But what I get for this price? Almost nothing. Only field validation make sense. Simple key-value mapping is about 50-70 lines of code, not a big deal, even for small project. Riak used mostly in web-based projects. Almost all modern projects use JSON. The simple workflow is get JSON from the client, validate, store in the database, query the database, send JSON to the client. If our mapper doesn't understand the JSON and cannot operate on it then it's not useful. We can use jsonpickle with a lot more profit. Just jsonpickle.encode the object and store it in Riak. I'm sorry, this is my position!

Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 9:01 AM, Andrey V. Martyanov <[hidden email]> wrote:
I'm just saying that to efficiently store and get the data user should have ability to customize the relationship. Riak has several options to get related data: secondary indexes, map-reduce etc. I think that the most important task of mapper is to validate the data and to track these dependencies. When you have enough meta information about objects storing these objects is not a big problem.


On Fri, Mar 2, 2012 at 8:39 AM, Shuhao Wu <[hidden email]> wrote:
The thing about riak-python-client is that it's already pretty high level, but not high level enough to be that simple.

From what I understand, tracking relationship between models will still involve a certain level of magic. Right now the magic is that when you save any document, all the related document will be saved into the database as well. You're saying that just tracking those relationship change is good enough and saving will be up to the user?

Shuhao


On Thu, Mar 1, 2012 at 10:20 PM, Andrey V. Martyanov <[hidden email]> wrote:
I'm not sure about forking the riak-python-client. The the whole idea of separation the low-level client library from the high-level mapper and the like is quite good. Some sort of SQLAlchemy does.

Best regards,
Andrey Martyanov


On Fri, Mar 2, 2012 at 8:15 AM, Shuhao Wu <[hidden email]> wrote:
Another thing I was considering is forking riak-python-client. Some of the stuff in there could use some fixing as well, and maybe just take out the core transports and such and plug that right into riakkit and skip the notion of RiakObject all together as those API is also somewhat weird. (such as add_link and remove_link)

Shuhao



On Thu, Mar 1, 2012 at 10:02 PM, Shuhao Wu <[hidden email]> wrote:
Hi!

First off, the code is LGPL licensed, not GPL. So you're free to put this into any code you want, it's just that when you want to modify riakkit, you have to release your modified version.

I just went over your idea and it would be a valid solution to some of the problems I'm having right now with my own projects with riakkit. When I initially started writing riakkit, I needed something to do RAD with, which would make sense to have an all-in-one, very simple to use framework, rather than working with serialization and what not. However, now I'm beginning to see problems in this approach when it comes to scalability as I'm slowing moving out of the RAD phase. I haven't done anything as I have not encountered any major issues yet (though I probably will).

So here's what I'm thinking for the next branch ( I still have to maintain the master branch due to some code i've written and need ): I'm going to break riakkit into 2 pieces. The first piece is the underlying piece, which will do what you said, serialize/deserialize objects, validation, track relationships and what not, however, it doesn't load it those references up nor does it try to accomplish everything all in one shot, and a RAD layer that's compatible with the current API that will allow RAD to work with it, but still reserve the flexibility of having that underlying API that's relatively easy (as oppose to dead simple) to work with and not as messy as what's going on right now.

I do agree that the code is messy.. Most of the time it's me fixing bugs I didn't anticipate as I go along and hacking everything into place..

One thing I'm not too clear about is what you mean by "Track dependencies between models". I'm not sure how you think I should go about approaching this (which right now is a pretty big disaster.. though relatively stable)

Cheers,

Shuhao

P.S. I don't want to go into the whole PEP8 discussion and 4 spaces vs whatever.. I just like the 2 spaces and the style guide that Google uses. Is that really a big deal if you're using it as a library?

On Thu, Mar 1, 2012 at 9:35 PM, Andrey V. Martyanov <[hidden email]> wrote:
Hi!

Two things force me to avoid using Riakkit: the code is GPL licensed and non pep8 compliant. So, I cannot fork it and make changes. The code needs clean up, for example, json module is imported in document.py, but not used. Riakkit has no conflict resolution logic, it's basically delegate the work to underlying python client. If I'd like to implement my save method I need to copy and rewrite more than 150 lines of code mess. What I think the mapper should do:
1. Serialize the objects to valid json;
2. Deserialize the data back to class instances;
3. Validate the fields data;
4. Track dependencies between models;

Simple example using Riakkit-like syntax;

class Post(Document):

    bucket_name  = 'posts'

    title = types.StringProperty(required=True)
    content = types.StringProperty(required=True)


class Comment(Document):

    bucket_name = 'comments'

    text = types.StringProperty(required=True)
    post = types.IndexReferenceProperty(reference_class=Post, collection_name='comments')

>>> post = Post('title', 'content')
>>> comment1 = ('comment1', post=post)
>>> comment2 = ('comment2', post=post)
>>> post.is_valid()
True
>> post.serialize()
{
    "key": "post_key",
    "title": "title",
    "content": "content",
    "comments": [
        {
            "key": "comment1_key",
            "text": "comment1"
        },
        {
            "key": "comment2_key",
            "text": "comment2"
        },
    ]
}
>>> post.store()
True
>>> post = Post.get_with_key('post_key')
>>> post.comments
['Post comment1', 'Post comment2']
>>> post_json_data = get_json_data_from_request()
>>> post = Post.deserialize(post_json_data)
>>> post.title
'title'
>>> post.comments
['Post comment1', 'Post comment2']

Something like that.


Best regards,
Andrey Martyanov

On Fri, Mar 2, 2012 at 5:39 AM, Sean Carey <[hidden email]> wrote:
Keep up the great work Shuhao!


On Mar 1, 2012, at 7:36 PM, Shuhao Wu <[hidden email]> wrote:

Hey guys,

I started writing riakkit approximately 3 months ago and announced it here. In case you missed that, Riakkit is basically an object mapper for python and riak, which seemed to be lacking in at the time. Now that 3 months has passed and I have developed this project into much more than what I had envisioned to begin with. It's currently also being used and tested in a project of mine (web application) as well and functioning quite nicely. I would say that currently this is the only object mapper that can compete with their counterparts in other languages for Riak in Python, although a lot of work still needs to be done before it could fully compete.

So today I felt that riakkit is "stable" and "clean" enough to deserve the first beta release. So I whipped up a website and it's released to pypi. Here's (again), a demo (the full tutorial can be found on the links below)

class User(Document):
    client = riak.RiakClient()
    bucket_name = "users"

    name = StringProperty(required=True)

>>> user = User(name="foo")
>>> user.save()
>>> print user.name
foo

If anyone is interested, you can find the website here: http://ultimatebuster.github.com/riakkit and the code at http://github.com/ultimatebuster/riakkit

Cheers!

Shuhao
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com








_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




--
-Michael


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
12