mr_queue gone wild

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

mr_queue gone wild

Sylvain Niles
We recently started trying to move our production environment over to
riak and we're seeing some weird behavior that's preventing us from
doing so:

Running: riak HEAD from github as of last Friday, riak_search turned
on with indexing of the problem bucket "events".
When we turn on our processes that start creating objects in the
bucket "events", the mr_queue directory starts growing massively and
the riak process starts spending most of its time in io_wait. With a
total of about 7000 objects (each is a ripple document that's got
maybe 10-20 lines of text in it) in the events bucket out bitcask dir
was ~240MB and the mr_queue dir was 1.2GB. Something is going horribly
wrong.. Our logs only show flow_timeouts for normal requests trying to
do simple map/reduce lookups. Any ideas where to look?

Thanks in advance,
Sylvain

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: mr_queue gone wild

Mathias Meyer
Sylvain,

you should not be using riak HEAD for anything that's close to your production environment. Development in Riak is in big flux right now, and it'll be hard for us to help you find the specific problem.

Could you please install Riak Search 0.14.2, best with a fresh installation, and try running this setup again to see if you get the same erroneous results? If you do, some more details on your data and the MapReduce jobs you're running would be great to reproduce and figure out the problem.

Mathias Meyer
Developer Advocate, Basho Technologies


On Mittwoch, 29. Juni 2011 at 00:41, Sylvain Niles wrote:

> We recently started trying to move our production environment over to
> riak and we're seeing some weird behavior that's preventing us from
> doing so:
>
> Running: riak HEAD from github as of last Friday, riak_search turned
> on with indexing of the problem bucket "events".
> When we turn on our processes that start creating objects in the
> bucket "events", the mr_queue directory starts growing massively and
> the riak process starts spending most of its time in io_wait. With a
> total of about 7000 objects (each is a ripple document that's got
> maybe 10-20 lines of text in it) in the events bucket out bitcask dir
> was ~240MB and the mr_queue dir was 1.2GB. Something is going horribly
> wrong.. Our logs only show flow_timeouts for normal requests trying to
> do simple map/reduce lookups. Any ideas where to look?
>
> Thanks in advance,
> Sylvain
>
> _______________________________________________
> riak-users mailing list
> [hidden email] (mailto:[hidden email])
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: mr_queue gone wild

Sylvain Niles
So I backrev'd everything to: Erlang R13B04, Riak 0.14.2 (no riak
search) and got rid of any functionality using search. After importing
the 7k objects the bitcask dir is ~41MB. Starting up our app
everything works fine until a worker starts updating objects with new
values at the rate of about 1-2/second. It finds those objects via a
map function that looks for one json integer and compares it with an
input (currently with a javascript function but I'm slowly porting
them all to erlang). While this worker is running the mr_queue
directory grows at about 1MB every 2 minutes, forever. It's my
understanding that pending m/r jobs are persisted to disk in this
directory, but the amount of work is trivial and the mr_queue never
gets smaller even after we shut down all our workers and leave riak
alone.

Is there a way to list the m/r jobs in the queue in case there's
something else going on? Is there a reason they never get removed?

Thanks in advance,
Sylvain


On Wed, Jun 29, 2011 at 12:59 AM, Mathias Meyer <[hidden email]> wrote:

> Sylvain,
>
> you should not be using riak HEAD for anything that's close to your production environment. Development in Riak is in big flux right now, and it'll be hard for us to help you find the specific problem.
>
> Could you please install Riak Search 0.14.2, best with a fresh installation, and try running this setup again to see if you get the same erroneous results? If you do, some more details on your data and the MapReduce jobs you're running would be great to reproduce and figure out the problem.
>
> Mathias Meyer
> Developer Advocate, Basho Technologies
>
>
> On Mittwoch, 29. Juni 2011 at 00:41, Sylvain Niles wrote:
>
>> We recently started trying to move our production environment over to
>> riak and we're seeing some weird behavior that's preventing us from
>> doing so:
>>
>> Running: riak HEAD from github as of last Friday, riak_search turned
>> on with indexing of the problem bucket "events".
>> When we turn on our processes that start creating objects in the
>> bucket "events", the mr_queue directory starts growing massively and
>> the riak process starts spending most of its time in io_wait. With a
>> total of about 7000 objects (each is a ripple document that's got
>> maybe 10-20 lines of text in it) in the events bucket out bitcask dir
>> was ~240MB and the mr_queue dir was 1.2GB. Something is going horribly
>> wrong.. Our logs only show flow_timeouts for normal requests trying to
>> do simple map/reduce lookups. Any ideas where to look?
>>
>> Thanks in advance,
>> Sylvain
>>
>> _______________________________________________
>> riak-users mailing list
>> [hidden email] (mailto:[hidden email])
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: mr_queue gone wild

Aphyr
The mr_queue is a bitcask, so you should expect it to grow monotonically
until compaction. The file size is not an indication of the number of
pending jobs. You can read the contents using any bitcask utility. For
example, using https://github.com/aphyr/bitcask-ruby:

$ bitcask --no-riak /var/lib/riak/mr_queue/ all
...
3264
bitcask_tombstone
3265
bitcask_tombstone
3266
bitcask_tombstone
3267
bitcask_tombstone

--Kyle

On 06/30/2011 01:52 PM, Sylvain Niles wrote:

> So I backrev'd everything to: Erlang R13B04, Riak 0.14.2 (no riak
> search) and got rid of any functionality using search. After importing
> the 7k objects the bitcask dir is ~41MB. Starting up our app
> everything works fine until a worker starts updating objects with new
> values at the rate of about 1-2/second. It finds those objects via a
> map function that looks for one json integer and compares it with an
> input (currently with a javascript function but I'm slowly porting
> them all to erlang). While this worker is running the mr_queue
> directory grows at about 1MB every 2 minutes, forever. It's my
> understanding that pending m/r jobs are persisted to disk in this
> directory, but the amount of work is trivial and the mr_queue never
> gets smaller even after we shut down all our workers and leave riak
> alone.
>
> Is there a way to list the m/r jobs in the queue in case there's
> something else going on? Is there a reason they never get removed?
>
> Thanks in advance,
> Sylvain
>
>
> On Wed, Jun 29, 2011 at 12:59 AM, Mathias Meyer<[hidden email]>  wrote:
>> Sylvain,
>>
>> you should not be using riak HEAD for anything that's close to your production environment. Development in Riak is in big flux right now, and it'll be hard for us to help you find the specific problem.
>>
>> Could you please install Riak Search 0.14.2, best with a fresh installation, and try running this setup again to see if you get the same erroneous results? If you do, some more details on your data and the MapReduce jobs you're running would be great to reproduce and figure out the problem.
>>
>> Mathias Meyer
>> Developer Advocate, Basho Technologies
>>
>>
>> On Mittwoch, 29. Juni 2011 at 00:41, Sylvain Niles wrote:
>>
>>> We recently started trying to move our production environment over to
>>> riak and we're seeing some weird behavior that's preventing us from
>>> doing so:
>>>
>>> Running: riak HEAD from github as of last Friday, riak_search turned
>>> on with indexing of the problem bucket "events".
>>> When we turn on our processes that start creating objects in the
>>> bucket "events", the mr_queue directory starts growing massively and
>>> the riak process starts spending most of its time in io_wait. With a
>>> total of about 7000 objects (each is a ripple document that's got
>>> maybe 10-20 lines of text in it) in the events bucket out bitcask dir
>>> was ~240MB and the mr_queue dir was 1.2GB. Something is going horribly
>>> wrong.. Our logs only show flow_timeouts for normal requests trying to
>>> do simple map/reduce lookups. Any ideas where to look?
>>>
>>> Thanks in advance,
>>> Sylvain
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> [hidden email] (mailto:[hidden email])
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: mr_queue gone wild

bryan-basho
Administrator
In reply to this post by Sylvain Niles
On Thu, Jun 30, 2011 at 4:52 PM, Sylvain Niles <[hidden email]> wrote:
> Is there a way to list the m/r jobs in the queue in case there's
> something else going on? Is there a reason they never get removed?

Hi, Sylvain.  As Aphyr noted, the mr_queue is a bitcask.  Because
bitcask is append-only storage, its size alone does not give a good
indication of the active dataset.  More specifically, unless you have
tweaked your bitcask parameters, you can expect the mr_queue directory
to grow to at least 2GB before purging unused data.  This is normal
and expected.

A few things worth noting:

 - The mr_queue is only used for Javascript MapReduce phases.  Erlang
   phases never touch it.  This is because there are a limited number
   of Javascript VMs available on a node, and all vnodes compete for
   them.  The mr_queue provides a place to offload the backlog of
   pending requeusts for Javascript interpreters, rather than keeping
   them in memory.

 - To check the depth of the mr_queue, connect to any Riak node's
   console (bin/riak attach), and call
   riak_kv_map_master:queue_depth/0.  It should show you the active
   depth of the mr_queue on each node in the cluster:

   $ bin/riak attach
   (dev1@127.0.0.1)> riak_kv_map_master:queue_depth().
   [{'dev1@127.0.0.1',0},
    {'dev2@127.0.0.1',0},
    {'dev3@127.0.0.1',0},
    {'dev4@127.0.0.1',0}]

   A result like the above means there are no pending map requests
   waiting.  A result like the following means there are 242 map
   requests waiting on the dev1 node, 128 on dev2, etc.:

   [{'dev1@127.0.0.1',242},
    {'dev2@127.0.0.1',128},
    {'dev3@127.0.0.1',212},
    {'dev4@127.0.0.1',230}]

   These numbers may be slightly confusing because a single logical
   map phase gets split into a separate map request for each vnode.
   So, you may have only one MapReduce request outstanding, but see
   numbers greater than 1 in this output.

Hope this helps,

Bryan Fink
Senior Software Developer,
Basho Technologies

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: mr_queue gone wild

Sylvain Niles
Thanks again for all the pointers. After much digging I found the
issue. We have UTF8 validation on the most data we put into riak (our
events bucket), but weren't validating UTF8 on some objects that are
linked to by many other objects (group bucket). Consequently some
malformed UTF8 (that displays fine via console, it's only spidermonkey
that has a problem) causes spidermonkey to die a horrible death when
it pulls an object that has a bad link. Does anyone have some example
code for returning a list of keys that have a common link? In this
case we only had one-way links.
Example:

Object1  ---link--->  BadUTF8.Object
Object2  ---link--->  BadUTF8.Object

I'll have to do this via the erlang client as doing it in Javascript
crashes every time :(

Any ideas?

PS: Is there any plan to make bad UTF8 not a riak-killer? This is the
second time it's bitten us and all of the examples we've seen don't
cause ruby or erlang any problems, only spidermonkey.

Bad UTF8: <<"Afro-Belly Boogie®  Fitness and Wellness-1800400">>


-Sylvain


On Tue, Jul 5, 2011 at 12:37 PM, Bryan Fink <[hidden email]> wrote:

> On Thu, Jun 30, 2011 at 4:52 PM, Sylvain Niles <[hidden email]> wrote:
>> Is there a way to list the m/r jobs in the queue in case there's
>> something else going on? Is there a reason they never get removed?
>
> Hi, Sylvain.  As Aphyr noted, the mr_queue is a bitcask.  Because
> bitcask is append-only storage, its size alone does not give a good
> indication of the active dataset.  More specifically, unless you have
> tweaked your bitcask parameters, you can expect the mr_queue directory
> to grow to at least 2GB before purging unused data.  This is normal
> and expected.
>
> A few things worth noting:
>
>  - The mr_queue is only used for Javascript MapReduce phases.  Erlang
>   phases never touch it.  This is because there are a limited number
>   of Javascript VMs available on a node, and all vnodes compete for
>   them.  The mr_queue provides a place to offload the backlog of
>   pending requeusts for Javascript interpreters, rather than keeping
>   them in memory.
>
>  - To check the depth of the mr_queue, connect to any Riak node's
>   console (bin/riak attach), and call
>   riak_kv_map_master:queue_depth/0.  It should show you the active
>   depth of the mr_queue on each node in the cluster:
>
>   $ bin/riak attach
>   (dev1@127.0.0.1)> riak_kv_map_master:queue_depth().
>   [{'dev1@127.0.0.1',0},
>    {'dev2@127.0.0.1',0},
>    {'dev3@127.0.0.1',0},
>    {'dev4@127.0.0.1',0}]
>
>   A result like the above means there are no pending map requests
>   waiting.  A result like the following means there are 242 map
>   requests waiting on the dev1 node, 128 on dev2, etc.:
>
>   [{'dev1@127.0.0.1',242},
>    {'dev2@127.0.0.1',128},
>    {'dev3@127.0.0.1',212},
>    {'dev4@127.0.0.1',230}]
>
>   These numbers may be slightly confusing because a single logical
>   map phase gets split into a separate map request for each vnode.
>   So, you may have only one MapReduce request outstanding, but see
>   numbers greater than 1 in this output.
>
> Hope this helps,
>
> Bryan Fink
> Senior Software Developer,
> Basho Technologies
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com