Invoking Python script as part of a M/R job?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Invoking Python script as part of a M/R job?

jeffrey k eliasen
I would like to write a job that invokes a python script to execute the processing of each node in a bucket. I can't find a way to do this using Javascript and I don't really know Erlang well enough to make this work... is there a sample piece of code somewhere that demonstrates this?

Thanks in advance!

jeff

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Invoking Python script as part of a M/R job?

Luke Bakken
Hi Jeff,

Erlang and Javascript are the only two supported "in process"
languages for map/reduce. Can you explain the process or provide the
python you want to use and perhaps someone on the list could help out
translating it to erlang?
--
Luke Bakken
CSE
[hidden email]


On Wed, Sep 25, 2013 at 4:47 PM, Jeffrey Eliasen <[hidden email]> wrote:

> I would like to write a job that invokes a python script to execute the
> processing of each node in a bucket. I can't find a way to do this using
> Javascript and I don't really know Erlang well enough to make this work...
> is there a sample piece of code somewhere that demonstrates this?
>
> Thanks in advance!
>
> jeff
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Invoking Python script as part of a M/R job?

jeffrey k eliasen
I'm trying to do some image processing using OpenCV. Later I'll be doing some video processing as well. In a future project I will be using R to do deep analysis on some data I'm collecting. In all these cases, what I want to do is very simple with external languages but very hard with both Erlang and Javascript.

What I want to do is simply invoke an external script on each element in a bucket in the general case so that I can use advanced external tools in an arbitrary manner. I was told by someone at Basho a long time ago (about a year, which is a long time in internet years) that this could be done by invoking scripts from Erlang, but I haven't heard back from him since then and was hoping someone on the list could point me at an example demonstrating this.

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

On Sep 26, 2013, at 10:41 , Luke Bakken <[hidden email]> wrote:

Hi Jeff,

Erlang and Javascript are the only two supported "in process"
languages for map/reduce. Can you explain the process or provide the
python you want to use and perhaps someone on the list could help out
translating it to erlang?
--
Luke Bakken
CSE
[hidden email]


On Wed, Sep 25, 2013 at 4:47 PM, Jeffrey Eliasen <[hidden email]> wrote:
I would like to write a job that invokes a python script to execute the
processing of each node in a bucket. I can't find a way to do this using
Javascript and I don't really know Erlang well enough to make this work...
is there a sample piece of code somewhere that demonstrates this?

Thanks in advance!

jeff

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Invoking Python script as part of a M/R job?

Toby Corkindale
On 26/09/13 14:33, jeffrey k eliasen wrote:

> I'm trying to do some image processing using OpenCV. Later I'll be doing
> some video processing as well. In a future project I will be using R to
> do deep analysis on some data I'm collecting. In all these cases, what I
> want to do is very simple with external languages but very hard with
> both Erlang and Javascript.
>
> What I want to do is simply invoke an external script on each element in
> a bucket in the general case so that I can use advanced external tools
> in an arbitrary manner. I was told by someone at Basho a long time ago
> (about a year, which is a long time in internet years) that this could
> be done by invoking scripts from Erlang, but I haven't heard back from
> him since then and was hoping someone on the list could point me at an
> example demonstrating this.

You'll make your life a lot easier if you invert your system.
Have your python scripts run somewhere and have them query Riak for keys
and data. Use a map-reduce job to partition the keys per script.
(eg. If you are running six python scripts in parallel, then you only
want a sixth of all keys going to each one. So they want to MR for keys
where modulo the key id == script id)

If you're looking for something to distribute your python scripts across
a number of compute nodes, then there are various existing systems for
it - Condor, Helios, Gearman, etc.


My own experiences of trying to use Riak as a map/reduce holistic system
over bulk data did not end well. It's really not designed to be that.

-Toby

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Invoking Python script as part of a M/R job?

Russell Brown
In reply to this post by jeffrey k eliasen
Not that it is something that answers your immediate need, I just thought I'd point you at this post where Brian Lee Yung Rowe attempts to integrate Riak MapReduce and R.

http://cartesianfaith.com/2011/08/17/teaser-running-r-as-a-mapreduce-job-from-riak/

On 26 Sep 2013, at 05:33, jeffrey k eliasen <[hidden email]> wrote:

> I'm trying to do some image processing using OpenCV. Later I'll be doing some video processing as well. In a future project I will be using R to do deep analysis on some data I'm collecting. In all these cases, what I want to do is very simple with external languages but very hard with both Erlang and Javascript.
>
> What I want to do is simply invoke an external script on each element in a bucket in the general case so that I can use advanced external tools in an arbitrary manner. I was told by someone at Basho a long time ago (about a year, which is a long time in internet years) that this could be done by invoking scripts from Erlang, but I haven't heard back from him since then and was hoping someone on the list could point me at an example demonstrating this.
>
> --
>
> jeffrey k eliasen
>
> Find and follow me on:
> Blog: http://jeff.jke.net
> Twitter: http://twitter.com/jeffreyeliasen
> Facebook: http://facebook.com/jeffrey.eliasen
>
> On Sep 26, 2013, at 10:41 , Luke Bakken <[hidden email]> wrote:
>
>> Hi Jeff,
>>
>> Erlang and Javascript are the only two supported "in process"
>> languages for map/reduce. Can you explain the process or provide the
>> python you want to use and perhaps someone on the list could help out
>> translating it to erlang?
>> --
>> Luke Bakken
>> CSE
>> [hidden email]
>>
>>
>> On Wed, Sep 25, 2013 at 4:47 PM, Jeffrey Eliasen <[hidden email]> wrote:
>>> I would like to write a job that invokes a python script to execute the
>>> processing of each node in a bucket. I can't find a way to do this using
>>> Javascript and I don't really know Erlang well enough to make this work...
>>> is there a sample piece of code somewhere that demonstrates this?
>>>
>>> Thanks in advance!
>>>
>>> jeff
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email]
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Invoking Python script as part of a M/R job?

jeffrey k eliasen
Hmmm, that's interesting. I don't quite understand why it works, but it's a step in the right direction. It surprises me that there isn't an easy way to invoke an external script from within a job.

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

On Sep 26, 2013, at 17:41 , Russell Brown <[hidden email]> wrote:

Not that it is something that answers your immediate need, I just thought I'd point you at this post where Brian Lee Yung Rowe attempts to integrate Riak MapReduce and R.

http://cartesianfaith.com/2011/08/17/teaser-running-r-as-a-mapreduce-job-from-riak/

On 26 Sep 2013, at 05:33, jeffrey k eliasen <[hidden email]> wrote:

I'm trying to do some image processing using OpenCV. Later I'll be doing some video processing as well. In a future project I will be using R to do deep analysis on some data I'm collecting. In all these cases, what I want to do is very simple with external languages but very hard with both Erlang and Javascript.

What I want to do is simply invoke an external script on each element in a bucket in the general case so that I can use advanced external tools in an arbitrary manner. I was told by someone at Basho a long time ago (about a year, which is a long time in internet years) that this could be done by invoking scripts from Erlang, but I haven't heard back from him since then and was hoping someone on the list could point me at an example demonstrating this.

--

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

On Sep 26, 2013, at 10:41 , Luke Bakken <[hidden email]> wrote:

Hi Jeff,

Erlang and Javascript are the only two supported "in process"
languages for map/reduce. Can you explain the process or provide the
python you want to use and perhaps someone on the list could help out
translating it to erlang?
--
Luke Bakken
CSE
[hidden email]


On Wed, Sep 25, 2013 at 4:47 PM, Jeffrey Eliasen <[hidden email]> wrote:
I would like to write a job that invokes a python script to execute the
processing of each node in a bucket. I can't find a way to do this using
Javascript and I don't really know Erlang well enough to make this work...
is there a sample piece of code somewhere that demonstrates this?

Thanks in advance!

jeff

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Invoking Python script as part of a M/R job?

jeffrey k eliasen
I suppose I could just use os:cmd from Erlang, but I've never worked with this before: http://www.erlang.org/doc/man/os.html. Does anyone know if this should be expected to work?

Thanks!

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

On Sep 26, 2013, at 20:01 , jeffrey k eliasen <[hidden email]> wrote:

Hmmm, that's interesting. I don't quite understand why it works, but it's a step in the right direction. It surprises me that there isn't an easy way to invoke an external script from within a job.

-- 

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

On Sep 26, 2013, at 17:41 , Russell Brown <[hidden email]> wrote:

Not that it is something that answers your immediate need, I just thought I'd point you at this post where Brian Lee Yung Rowe attempts to integrate Riak MapReduce and R.

http://cartesianfaith.com/2011/08/17/teaser-running-r-as-a-mapreduce-job-from-riak/

On 26 Sep 2013, at 05:33, jeffrey k eliasen <[hidden email]> wrote:

I'm trying to do some image processing using OpenCV. Later I'll be doing some video processing as well. In a future project I will be using R to do deep analysis on some data I'm collecting. In all these cases, what I want to do is very simple with external languages but very hard with both Erlang and Javascript.

What I want to do is simply invoke an external script on each element in a bucket in the general case so that I can use advanced external tools in an arbitrary manner. I was told by someone at Basho a long time ago (about a year, which is a long time in internet years) that this could be done by invoking scripts from Erlang, but I haven't heard back from him since then and was hoping someone on the list could point me at an example demonstrating this.

--

jeffrey k eliasen

Find and follow me on:
Blog: http://jeff.jke.net
Twitter: http://twitter.com/jeffreyeliasen
Facebook: http://facebook.com/jeffrey.eliasen

On Sep 26, 2013, at 10:41 , Luke Bakken <[hidden email]> wrote:

Hi Jeff,

Erlang and Javascript are the only two supported "in process"
languages for map/reduce. Can you explain the process or provide the
python you want to use and perhaps someone on the list could help out
translating it to erlang?
--
Luke Bakken
CSE
[hidden email]


On Wed, Sep 25, 2013 at 4:47 PM, Jeffrey Eliasen <[hidden email]> wrote:
I would like to write a job that invokes a python script to execute the
processing of each node in a bucket. I can't find a way to do this using
Javascript and I don't really know Erlang well enough to make this work...
is there a sample piece of code somewhere that demonstrates this?

Thanks in advance!

jeff

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com