Upgraded riak 1.4.9 is pegging the CPU

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgraded riak 1.4.9 is pegging the CPU

Alain Rodriguez
Hi all,

I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to 1.4.9. The rest are running 1.4.0.

Ever since I am seeing the upgraded node, riak01 consuming a significantly larger percent of CPU and the PUT times on it have gotten worse. htop indicicates one particular process pegging the CPU, and many many more processes running than I was used to seeing before.

Has anyone seen this before? Do I have to retune something for 1.4.9?

I am attaching htop, cpu and put graphs, and my app.config used across all servers.

Thanks!



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

app.config.erb (19K) Download Attachment
vm.args.erb (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded riak 1.4.9 is pegging the CPU

Shane McEwan-2
On 05/06/14 16:20, Alain Rodriguez wrote:
> Hi all,
>
> I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to
> 1.4.9. The rest are running 1.4.0.
>
> Ever since I am seeing the upgraded node, riak01 consuming a
> significantly larger percent of CPU and the PUT times on it have gotten
> worse. htop indicicates one particular process pegging the CPU, and many
> many more processes running than I was used to seeing before.

G'day!

Did you turn off and remove the Active Anti Entropy files before upgrading?

From the 1.4.8 release notes:

IMPORTANT We recommend removing current AAE trees before upgrading. That
is, all files under the anti_entropy sub-directory. This will avoid
potentially large amounts of repair activity once correct hashes start
being added. The data in the current trees can only be fixed by a full
rebuild, so this repair activity is wasteful. Trees will start to build
once AAE is re-enabled. To minimize the impact of this, we recommend
upgrading during a period of low activity.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded riak 1.4.9 is pegging the CPU

Alain Rodriguez
Thanks for the quick reply and no I did not. Is this something I should be able to do now (stop, remove files, start again) or is it too late? How could I verify this is the issue?


On Thu, Jun 5, 2014 at 8:42 AM, Shane McEwan <[hidden email]> wrote:
On 05/06/14 16:20, Alain Rodriguez wrote:
> Hi all,
>
> I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to
> 1.4.9. The rest are running 1.4.0.
>
> Ever since I am seeing the upgraded node, riak01 consuming a
> significantly larger percent of CPU and the PUT times on it have gotten
> worse. htop indicicates one particular process pegging the CPU, and many
> many more processes running than I was used to seeing before.

G'day!

Did you turn off and remove the Active Anti Entropy files before upgrading?

From the 1.4.8 release notes:

IMPORTANT We recommend removing current AAE trees before upgrading. That
is, all files under the anti_entropy sub-directory. This will avoid
potentially large amounts of repair activity once correct hashes start
being added. The data in the current trees can only be fixed by a full
rebuild, so this repair activity is wasteful. Trees will start to build
once AAE is re-enabled. To minimize the impact of this, we recommend
upgrading during a period of low activity.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded riak 1.4.9 is pegging the CPU

Alain Rodriguez
Actually I just noticed it is likely the AAE issue:

2014-06-05 14:53:47.587 [error] <0.16054.31> CRASH REPORT Process <0.16054.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1061872283373234151507364761270424381468763488256/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328
2014-06-05 14:53:47.588 [error] <0.16056.31> CRASH REPORT Process <0.16056.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1335903840372778448670555667404727447654250840064/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328
2014-06-05 14:53:47.588 [error] <0.16055.31> CRASH REPORT Process <0.16055.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1267395951122892374379757940871151681107879002112/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328

Bollocks!


On Thu, Jun 5, 2014 at 8:49 AM, Alain Rodriguez <[hidden email]> wrote:
Thanks for the quick reply and no I did not. Is this something I should be able to do now (stop, remove files, start again) or is it too late? How could I verify this is the issue?


On Thu, Jun 5, 2014 at 8:42 AM, Shane McEwan <[hidden email]> wrote:
On 05/06/14 16:20, Alain Rodriguez wrote:
> Hi all,
>
> I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to
> 1.4.9. The rest are running 1.4.0.
>
> Ever since I am seeing the upgraded node, riak01 consuming a
> significantly larger percent of CPU and the PUT times on it have gotten
> worse. htop indicicates one particular process pegging the CPU, and many
> many more processes running than I was used to seeing before.

G'day!

Did you turn off and remove the Active Anti Entropy files before upgrading?

From the 1.4.8 release notes:

IMPORTANT We recommend removing current AAE trees before upgrading. That
is, all files under the anti_entropy sub-directory. This will avoid
potentially large amounts of repair activity once correct hashes start
being added. The data in the current trees can only be fixed by a full
rebuild, so this repair activity is wasteful. Trees will start to build
once AAE is re-enabled. To minimize the impact of this, we recommend
upgrading during a period of low activity.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded riak 1.4.9 is pegging the CPU

Engel Sanchez-2
Hi Alain. I don't think you are seeing the AAE issue. The problem with upgrading from 1.4.4-1.4.7 to 1.4.8 was a broken hash function in those, which made the AAE trees incompatible. You should not have the same problem in 1.4.0.  It seems that Erlang processes are repeatedly crashing and restarting. It would be good to grab all your logs before they rotate so we can take a look at exactly what is the first thing crashing and causing this snowball effect.


On Thu, Jun 5, 2014 at 11:58 AM, Alain Rodriguez <[hidden email]> wrote:
Actually I just noticed it is likely the AAE issue:

2014-06-05 14:53:47.587 [error] <0.16054.31> CRASH REPORT Process <0.16054.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1061872283373234151507364761270424381468763488256/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328
2014-06-05 14:53:47.588 [error] <0.16056.31> CRASH REPORT Process <0.16056.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1335903840372778448670555667404727447654250840064/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328
2014-06-05 14:53:47.588 [error] <0.16055.31> CRASH REPORT Process <0.16055.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1267395951122892374379757940871151681107879002112/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328

Bollocks!


On Thu, Jun 5, 2014 at 8:49 AM, Alain Rodriguez <[hidden email]> wrote:
Thanks for the quick reply and no I did not. Is this something I should be able to do now (stop, remove files, start again) or is it too late? How could I verify this is the issue?


On Thu, Jun 5, 2014 at 8:42 AM, Shane McEwan <[hidden email]> wrote:
On 05/06/14 16:20, Alain Rodriguez wrote:
> Hi all,
>
> I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to
> 1.4.9. The rest are running 1.4.0.
>
> Ever since I am seeing the upgraded node, riak01 consuming a
> significantly larger percent of CPU and the PUT times on it have gotten
> worse. htop indicicates one particular process pegging the CPU, and many
> many more processes running than I was used to seeing before.

G'day!

Did you turn off and remove the Active Anti Entropy files before upgrading?

From the 1.4.8 release notes:

IMPORTANT We recommend removing current AAE trees before upgrading. That
is, all files under the anti_entropy sub-directory. This will avoid
potentially large amounts of repair activity once correct hashes start
being added. The data in the current trees can only be fixed by a full
rebuild, so this repair activity is wasteful. Trees will start to build
once AAE is re-enabled. To minimize the impact of this, we recommend
upgrading during a period of low activity.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Upgraded riak 1.4.9 is pegging the CPU

Engel Sanchez-2
Alain, thanks for the logs you sent me on the side.  I'm not yet sure what the root cause is, but I saw a lot of handoff activity and busy distributed port messages, which indicate the single TCP connection between two Erlang nodes is completely saturated.  Since there is too much going on, turning off AAE and examining your cluster with less activity might still be a good idea.  Check the output of riak-admin transfers until it is quiet.   I noticed you have a file limit of 8192. That is not low, but newer Riaks eat more file handles, so it would be a good idea to double that.  Let us know how the stats and the logs look like after AAE is off to see what else we can do.


On Thu, Jun 5, 2014 at 1:05 PM, Engel Sanchez <[hidden email]> wrote:
Hi Alain. I don't think you are seeing the AAE issue. The problem with upgrading from 1.4.4-1.4.7 to 1.4.8 was a broken hash function in those, which made the AAE trees incompatible. You should not have the same problem in 1.4.0.  It seems that Erlang processes are repeatedly crashing and restarting. It would be good to grab all your logs before they rotate so we can take a look at exactly what is the first thing crashing and causing this snowball effect.


On Thu, Jun 5, 2014 at 11:58 AM, Alain Rodriguez <[hidden email]> wrote:
Actually I just noticed it is likely the AAE issue:

2014-06-05 14:53:47.587 [error] <0.16054.31> CRASH REPORT Process <0.16054.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1061872283373234151507364761270424381468763488256/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328
2014-06-05 14:53:47.588 [error] <0.16056.31> CRASH REPORT Process <0.16056.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1335903840372778448670555667404727447654250840064/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328
2014-06-05 14:53:47.588 [error] <0.16055.31> CRASH REPORT Process <0.16055.31> with 0 neighbours exited with reason: no match of right hand value {error,{db_open,"IO error: lock /var/lib/riak/anti_entropy/1267395951122892374379757940871151681107879002112/LOCK: already held by process"}} in hashtree:new_segment_store/2 line 505 in gen_server:init_it/6 line 328

Bollocks!


On Thu, Jun 5, 2014 at 8:49 AM, Alain Rodriguez <[hidden email]> wrote:
Thanks for the quick reply and no I did not. Is this something I should be able to do now (stop, remove files, start again) or is it too late? How could I verify this is the issue?


On Thu, Jun 5, 2014 at 8:42 AM, Shane McEwan <[hidden email]> wrote:
On 05/06/14 16:20, Alain Rodriguez wrote:
> Hi all,
>
> I upgraded 1 of 9 riak nodes in a cluster last night from 1.4.0 to
> 1.4.9. The rest are running 1.4.0.
>
> Ever since I am seeing the upgraded node, riak01 consuming a
> significantly larger percent of CPU and the PUT times on it have gotten
> worse. htop indicicates one particular process pegging the CPU, and many
> many more processes running than I was used to seeing before.

G'day!

Did you turn off and remove the Active Anti Entropy files before upgrading?

From the 1.4.8 release notes:

IMPORTANT We recommend removing current AAE trees before upgrading. That
is, all files under the anti_entropy sub-directory. This will avoid
potentially large amounts of repair activity once correct hashes start
being added. The data in the current trees can only be fixed by a full
rebuild, so this repair activity is wasteful. Trees will start to build
once AAE is re-enabled. To minimize the impact of this, we recommend
upgrading during a period of low activity.

Shane.

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com