How Riak Handle Request?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How Riak Handle Request?

Jing Liu
Hi, I'm learning Riak. I'm wondering that if this is the case that on
each physical node, a single process is reading the connection,
decoding the buffer, spawning a new process to process the request and
then be able to handle next request from client?

Thanks,
Jing Liu

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Christopher Meiklejohn-2
On Sat, Apr 15, 2017 at 2:16 PM, Jing Liu <[hidden email]> wrote:
> Hi, I'm learning Riak. I'm wondering that if this is the case that on
> each physical node, a single process is reading the connection,
> decoding the buffer, spawning a new process to process the request and
> then be able to handle next request from client?

Hi Jing,

Depending on what you're looking it, it might be handled differently.
Typically, a process pool is used to handle incoming requests, but
processing of the request itself is handled by several processes that
operate concurrently.

What specific operation are you referring to?

Thanks,
Christopher

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Jing Liu
Hey,

I'm specifically concerned about connection reading, data sending,
data encoding and decoding. Is that done by a single process, a
fixed-size process pool or does Riak spawns a new process for every
incoming request/connection?

Thanks very much !
Jing

On Mon, Apr 17, 2017 at 12:36 PM, Christopher Meiklejohn
<[hidden email]> wrote:

> On Sat, Apr 15, 2017 at 2:16 PM, Jing Liu <[hidden email]> wrote:
>> Hi, I'm learning Riak. I'm wondering that if this is the case that on
>> each physical node, a single process is reading the connection,
>> decoding the buffer, spawning a new process to process the request and
>> then be able to handle next request from client?
>
> Hi Jing,
>
> Depending on what you're looking it, it might be handled differently.
> Typically, a process pool is used to handle incoming requests, but
> processing of the request itself is handled by several processes that
> operate concurrently.
>
> What specific operation are you referring to?
>
> Thanks,
> Christopher

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Fred Dushin
What do you mean by 'process'?

If you mean operating system process, then, No.  A Riak instance runs in a single OS process.  There is a Riak instance per "node" in the cluster, where a "node" is a machine, container, etc.  Probably what the OMG guys used to call a "capsule"  (cute)  There are typically many nodes in a Riak cluster; hence multiple OS processes, but in general it only makes sense to run each process on a separate node or machine, because things break.

If you mean a lightweight Erlang process, then yes, there is an Erlang process per connection.  From that point on, it gets ... complicated.  There are process corresponding to coordinating finite state machines, processes (plural) corresponding to vnodes, and there is a long list of processes that run support for all things between.  But all of these Erlang processes (or "actors") run in a single operating system process, the Erlang virtual machine, or abstract machine, if you learned how to computer before Java.

Does that help?

-Fred

> On Apr 17, 2017, at 9:19 PM, Jing Liu <[hidden email]> wrote:
>
> Hey,
>
> I'm specifically concerned about connection reading, data sending,
> data encoding and decoding. Is that done by a single process, a
> fixed-size process pool or does Riak spawns a new process for every
> incoming request/connection?
>
> Thanks very much !
> Jing
>
> On Mon, Apr 17, 2017 at 12:36 PM, Christopher Meiklejohn
> <[hidden email]> wrote:
>> On Sat, Apr 15, 2017 at 2:16 PM, Jing Liu <[hidden email]> wrote:
>>> Hi, I'm learning Riak. I'm wondering that if this is the case that on
>>> each physical node, a single process is reading the connection,
>>> decoding the buffer, spawning a new process to process the request and
>>> then be able to handle next request from client?
>>
>> Hi Jing,
>>
>> Depending on what you're looking it, it might be handled differently.
>> Typically, a process pool is used to handle incoming requests, but
>> processing of the request itself is handled by several processes that
>> operate concurrently.
>>
>> What specific operation are you referring to?
>>
>> Thanks,
>> Christopher
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Carlo Pires
Fred,

Are you saying that Riak doesn't start erlang runtime in SMP and a machine with multiple processes will have only one dedicated to each Riak instance?

2017-04-17 23:43 GMT-03:00 Fred Dushin <[hidden email]>:
What do you mean by 'process'?

If you mean operating system process, then, No.  A Riak instance runs in a single OS process.  There is a Riak instance per "node" in the cluster, where a "node" is a machine, container, etc.  Probably what the OMG guys used to call a "capsule"  (cute)  There are typically many nodes in a Riak cluster; hence multiple OS processes, but in general it only makes sense to run each process on a separate node or machine, because things break.

If you mean a lightweight Erlang process, then yes, there is an Erlang process per connection.  From that point on, it gets ... complicated.  There are process corresponding to coordinating finite state machines, processes (plural) corresponding to vnodes, and there is a long list of processes that run support for all things between.  But all of these Erlang processes (or "actors") run in a single operating system process, the Erlang virtual machine, or abstract machine, if you learned how to computer before Java.

Does that help?

-Fred

> On Apr 17, 2017, at 9:19 PM, Jing Liu <[hidden email]> wrote:
>
> Hey,
>
> I'm specifically concerned about connection reading, data sending,
> data encoding and decoding. Is that done by a single process, a
> fixed-size process pool or does Riak spawns a new process for every
> incoming request/connection?
>
> Thanks very much !
> Jing
>
> On Mon, Apr 17, 2017 at 12:36 PM, Christopher Meiklejohn
> <[hidden email]> wrote:
>> On Sat, Apr 15, 2017 at 2:16 PM, Jing Liu <[hidden email]> wrote:
>>> Hi, I'm learning Riak. I'm wondering that if this is the case that on
>>> each physical node, a single process is reading the connection,
>>> decoding the buffer, spawning a new process to process the request and
>>> then be able to handle next request from client?
>>
>> Hi Jing,
>>
>> Depending on what you're looking it, it might be handled differently.
>> Typically, a process pool is used to handle incoming requests, but
>> processing of the request itself is handled by several processes that
>> operate concurrently.
>>
>> What specific operation are you referring to?
>>
>> Thanks,
>> Christopher
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



--
  Carlo Pires

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

DeadZen
I don't see that being alluded to.. An erlang vm, which occupies an os
process runs an instance of the riak application, with a configurable
number of asynchronous threads for io and schedulers for processes
which are internal to erlang, this can be any number of hundreds of
thousands of "light processes". Erlang is inherently SMP friendly, as
processes hardly ever share memory, the non SMP portions are also
going away iirc. Erlangs os pid hosts many erlang processes, these
processes naturally are multiprocessor friendly, the schedulers should
be configured to align with your available processor cores.

On Tue, Apr 18, 2017 at 9:16 AM, Carlo Pires <[hidden email]> wrote:

> Fred,
>
> Are you saying that Riak doesn't start erlang runtime in SMP and a machine
> with multiple processes will have only one dedicated to each Riak instance?
>
> 2017-04-17 23:43 GMT-03:00 Fred Dushin <[hidden email]>:
>>
>> What do you mean by 'process'?
>>
>> If you mean operating system process, then, No.  A Riak instance runs in a
>> single OS process.  There is a Riak instance per "node" in the cluster,
>> where a "node" is a machine, container, etc.  Probably what the OMG guys
>> used to call a "capsule"  (cute)  There are typically many nodes in a Riak
>> cluster; hence multiple OS processes, but in general it only makes sense to
>> run each process on a separate node or machine, because things break.
>>
>> If you mean a lightweight Erlang process, then yes, there is an Erlang
>> process per connection.  From that point on, it gets ... complicated.  There
>> are process corresponding to coordinating finite state machines, processes
>> (plural) corresponding to vnodes, and there is a long list of processes that
>> run support for all things between.  But all of these Erlang processes (or
>> "actors") run in a single operating system process, the Erlang virtual
>> machine, or abstract machine, if you learned how to computer before Java.
>>
>> Does that help?
>>
>> -Fred
>>
>> > On Apr 17, 2017, at 9:19 PM, Jing Liu <[hidden email]> wrote:
>> >
>> > Hey,
>> >
>> > I'm specifically concerned about connection reading, data sending,
>> > data encoding and decoding. Is that done by a single process, a
>> > fixed-size process pool or does Riak spawns a new process for every
>> > incoming request/connection?
>> >
>> > Thanks very much !
>> > Jing
>> >
>> > On Mon, Apr 17, 2017 at 12:36 PM, Christopher Meiklejohn
>> > <[hidden email]> wrote:
>> >> On Sat, Apr 15, 2017 at 2:16 PM, Jing Liu <[hidden email]> wrote:
>> >>> Hi, I'm learning Riak. I'm wondering that if this is the case that on
>> >>> each physical node, a single process is reading the connection,
>> >>> decoding the buffer, spawning a new process to process the request and
>> >>> then be able to handle next request from client?
>> >>
>> >> Hi Jing,
>> >>
>> >> Depending on what you're looking it, it might be handled differently.
>> >> Typically, a process pool is used to handle incoming requests, but
>> >> processing of the request itself is handled by several processes that
>> >> operate concurrently.
>> >>
>> >> What specific operation are you referring to?
>> >>
>> >> Thanks,
>> >> Christopher
>> >
>> > _______________________________________________
>> > riak-users mailing list
>> > [hidden email]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
> --
>   Carlo Pires
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Christopher Meiklejohn-2
In reply to this post by Carlo Pires
On Tue, Apr 18, 2017 at 3:16 PM, Carlo Pires <[hidden email]> wrote:
> Fred,
>
> Are you saying that Riak doesn't start erlang runtime in SMP and a machine
> with multiple processes will have only one dedicated to each Riak instance?

Hi Carlo,

Riak is started in SMP mode, with a scheduler per thread, but running
within a single Unix process.

Thanks,
- Christopher

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Matthew Von-Maszewski
"with a scheduler per thread"

Slight correction:  with a scheduler per logical CPU core.

If you have other applications on the same box, e.g. java code, you might want to reduce the number of schedulers.  See https://github.com/basho/leveldb/wiki/riak-tuning-2 for more details

Matthew


On Apr 18, 2017, at 9:34 AM, Christopher Meiklejohn <[hidden email]> wrote:

On Tue, Apr 18, 2017 at 3:16 PM, Carlo Pires <[hidden email]> wrote:
Fred,

Are you saying that Riak doesn't start erlang runtime in SMP and a machine
with multiple processes will have only one dedicated to each Riak instance?

Hi Carlo,

Riak is started in SMP mode, with a scheduler per thread, but running
within a single Unix process.

Thanks,
- Christopher

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How Riak Handle Request?

Christopher Meiklejohn-2
On Tue, Apr 18, 2017 at 4:20 PM, Matthew Von-Maszewski
<[hidden email]> wrote:
> "with a scheduler per thread"
>
> Slight correction:  with a scheduler per logical CPU core.

Yes, you're right.  That was a typo.

Christopher

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Loading...