Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Zubair Quraishi
Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put objects into Riak fine on both platforms, but when I do list_buckets( ) it doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone else experienced this?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Rusty Klophaus
Hi Zubair,

Hmmm... that is strange. Can you provide the following information to help me troubleshoot?:

- Can you post the actual error message that you see?
- Are you using the same backend on each system? Which backend are you using?
- Approximately how many buckets have you created, and how many objects are stored in total?
- If you delete or rename the ./rel/riak/data directory and write more objects, can you get the timeout to repeat? If so, when do you start to see the problem?

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:30 AM, Zubair Quraishi <[hidden email]> wrote:
Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put objects into Riak fine on both platforms, but when I do list_buckets( ) it doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone else experienced this?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Rusty Klophaus
Hi Zubair,

Two more questions:

- Is this on a single-node or multi-node cluster?
- Can you get to the Riak console (start Riak with "./bin/riak console", run "io:format("~p", [riak_ring_manager:get_my_ring()])." and send me the output?

Finally, to clarify my instructions on deleting the "data" directory, you will need to shut down Riak, then rename or delete the ./rel/riak/data directory, and then restart Riak.

Thanks,
Rusty

On Wed, Feb 10, 2010 at 11:43 AM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Hmmm... that is strange. Can you provide the following information to help me troubleshoot?:

- Can you post the actual error message that you see?
- Are you using the same backend on each system? Which backend are you using?
- Approximately how many buckets have you created, and how many objects are stored in total?
- If you delete or rename the ./rel/riak/data directory and write more objects, can you get the timeout to repeat? If so, when do you start to see the problem?

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:30 AM, Zubair Quraishi <[hidden email]> wrote:
Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put objects into Riak fine on both platforms, but when I do list_buckets( ) it doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone else experienced this?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Zubair Quraishi
Yes, of course. Some extra info I forgot earlier is that I have two nodes, 192.168.1.3 and 192.168.1.5, and 192.168.1.5 is down.

- I'm using the default backend

- just 1 bucket I think (groceries :)

-  {ok, C} = riak:client_connect('[hidden email]').
{ok,{riak_client,'[hidden email]',<<0,38,76,114>>}}
([hidden email])2> C:list_buckets().

=ERROR REPORT==== 10-Feb-2010::17:49:58 ===
** State machine <5066.1052.0> terminating
** Last event in was timeout
** When State == waiting_kl
**      Data  == {state,<0.46.0>,plain,
                     {sbf,3.0e-5,0.85,1,0,
                         [{bloom,4.500000000000001e-6,11784498,24,0,
                              [{array,621379,0,0,1000000},
                               {array,621379,0,0,1000000},
                               {array,621379,0,0,1000000},
...
                               {array,621379,0,0,1000000},
                               {array,621379,0,0,1000000},
                               {array,621379,0,0,1000000}]}]},
                     [{45671926166590716193865151022383844364247891968,
                       '[hidden email]'},
                      {68507889249886074290797726533575766546371837952,
                       '[hidden email]'},
                      {91343852333181432387730302044767688728495783936,
                       '[hidden email]'},
                      {114179815416476790484662877555959610910619729920,
                       '[hidden email]'},
                      {137015778499772148581595453067151533092743675904,
                       '[hidden email]'},
                      {159851741583067506678528028578343455274867621888,
                       '[hidden email]'},
                      {182687704666362864775460604089535377456991567872,

...
                      {1050454301831586472458898473514828420377701515264,

                       '[hidden email]'},
                      {1347321821914426127719021955160323408745312813056,
                       '[hidden email]'},
                      {1370157784997721485815954530671515330927436759040,
                       '[hidden email]'},
                      {1392993748081016843912887106182707253109560705024,
                       '[hidden email]'},
                      {1415829711164312202009819681693899175291684651008,
                       '[hidden email]'},
                      {1438665674247607560106752257205091097473808596992,
                       '[hidden email]'}],
                     '_',120000,112621262,
                     {chstate,'[hidden email]',
                         [{'[hidden email]',{32,63432664825}}],
                         {64,
                          [{0,'[hidden email]'},
                           {22835963083295358096932575511191922182123945984,
                            '[hidden email]'},
                           {45671926166590716193865151022383844364247891968,
                            '[hidden email]'},
                           {68507889249886074290797726533575766546371837952,
                            '[hidden email]'},
                           {91343852333181432387730302044767688728495783936,
                            '[hidden email]'},
                           {114179815416476790484662877555959610910619729920,
                            '[hidden email]'},
                           {137015778499772148581595453067151533092743675904,
                            '[hidden email]'},
                           {159851741583067506678528028578343455274867621888,
                            '[hidden email]'},
...
                            '[hidden email]'},
                           {1415829711164312202009819681693899175291684651008,
                            '[hidden email]'},
                           {1438665674247607560106752257205091097473808596992,
                            '[hidden email]'}]},
                         {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}}}}
** Reason for termination =
** {function_clause,
       [{riak_keys_fsm,waiting_kl,
            [timeout,
             {state,<0.46.0>,plain,
                 {sbf,3.0e-5,0.85,1,0,
                     [{bloom,4.500000000000001e-6,11784498,24,0,
                          [{array,621379,0,0,1000000},
                           {array,621379,0,0,1000000},
                           {array,621379,0,0,1000000},
...
                           {array,621379,0,0,1000000},
                           {array,621379,0,0,1000000}]}]},
                 [{45671926166590716193865151022383844364247891968,
                   '[hidden email]'},
                  {68507889249886074290797726533575766546371837952,
                   '[hidden email]'},
...           
                  {1233142006497949337234359077604363797834693083136,
                   '[hidden email]'},
                  {1255977969581244695331291653115555720016817029120,
                   '[hidden email]'},
                  {1278813932664540053428224228626747642198940975104,
                   '[hidden email]'},
                  {1438665674247607560106752257205091097473808596992,
                   '[hidden email]'}],
                 '_',120000,112621262,
                 {chstate,'[hidden email]',
                     [{'[hidden email]',{32,63432664825}}],
                     {64,
                      [{0,'[hidden email]'},
                       {22835963083295358096932575511191922182123945984,
                        '[hidden email]'},
.......
                        '[hidden email]'},
                       {1392993748081016843912887106182707253109560705024,
                        '[hidden email]'},
                       {1415829711164312202009819681693899175291684651008,
                        '[hidden email]'},
                       {1438665674247607560106752257205091097473808596992,
                        '[hidden email]'}]},
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                           []}}}}}]},
        {gen_fsm,handle_msg,7},
        {proc_lib,init_p_do_apply,3}]}
** at node [hidden email] **


- Yes, deleting data/ring did work, so I guess when the second Riak node dropped it caused some problem?

- multi-node, but one node down

- users-Mac-Pro:riak root# ./bin/riak console
Exec: /opt/local/bin/riak/rel/riak/erts-5.7.4/bin/erlexec -boot /opt/local/bin/riak/rel/riak/releases/0.8/riak -embedded -config /opt/local/bin/riak/rel/riak/etc/app.config -args_file /opt/local/bin/riak/rel/riak/etc/vm.args -- console
Root: /opt/local/bin/riak/rel/riak
Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:5] [hipe] [kernel-poll:true]

** Bad path can't read /opt/local/bin/riak/rel/riak/lib/kernel-2.13.4.ez/kernel-2.13.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/stdlib-1.16.4.ez/stdlib-1.16.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/sasl-2.1.8.ez/sasl-2.1.8/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/crypto-1.6.3.ez/crypto-1.6.3/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/runtime_tools-1.8.2.ez/runtime_tools-1.8.2/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/mochiweb-0.01.ez/mochiweb-0.01/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/webmachine-1.5.2.ez/webmachine-1.5.2/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/riak-0.7.1.ez/riak-0.7.1/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/xmerl-1.2.3.ez/xmerl-1.2.3/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/webtool-0.8.5.ez/webtool-0.8.5/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/tools-2.6.5.ez/tools-2.6.5/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/syntax_tools-1.6.4.ez/syntax_tools-1.6.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/ssl-3.10.7.ez/ssl-3.10.7/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/snmp-4.15.ez/snmp-4.15/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/public_key-0.4.ez/public_key-0.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/protobuffs-4.ez/protobuffs-4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/otp_mibs-1.0.6.ez/otp_mibs-1.0.6/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/os_mon-2.2.4.ez/os_mon-2.2.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/observer-0.9.8.1.ez/observer-0.9.8.1/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/mnesia-4.4.12.ez/mnesia-4.4.12/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/inets-5.2.ez/inets-5.2/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/hipe-3.7.4.ez/hipe-3.7.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/gs-1.5.11.ez/gs-1.5.11/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/eunit-2.1.4.ez/eunit-2.1.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/et-1.3.3.ez/et-1.3.3/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/erts-5.7.4.ez/erts-5.7.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/edoc-0.7.6.5.ez/edoc-0.7.6.5/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/compiler-4.6.4.ez/compiler-4.6.4/ebin
** Found 0 name clashes in code paths
Eshell V5.7.4  (abort with ^G)
([hidden email])1>


([hidden email])1> io:format("~p", [riak_ring_manager:get_my_ring()]).
{ok,{chstate,'[hidden email]',[],
             {64,
              [{0,'[hidden email]'},
               {22835963083295358096932575511191922182123945984,
                '[hidden email]'},
               {45671926166590716193865151022383844364247891968,
                '[hidden email]'},
               {68507889249886074290797726533575766546371837952,
                '[hidden email]'},
               {91343852333181432387730302044767688728495783936,
                '[hidden email]'},
               {114179815416476790484662877555959610910619729920,
                '[hidden email]'},
               {137015778499772148581595453067151533092743675904,
                '[hidden email]'},
               {159851741583067506678528028578343455274867621888,
                '[hidden email]'},
               {182687704666362864775460604089535377456991567872,
                '[hidden email]'},
               {205523667749658222872393179600727299639115513856,
                '[hidden email]'},
               {228359630832953580969325755111919221821239459840,
                '[hidden email]'},
               {251195593916248939066258330623111144003363405824,
                '[hidden email]'},
               {274031556999544297163190906134303066185487351808,
                '[hidden email]'},
               {296867520082839655260123481645494988367611297792,
                '[hidden email]'},
               {319703483166135013357056057156686910549735243776,
                '[hidden email]'},
               {342539446249430371453988632667878832731859189760,
                '[hidden email]'},
               {365375409332725729550921208179070754913983135744,
                '[hidden email]'},
               {388211372416021087647853783690262677096107081728,
                '[hidden email]'},
               {411047335499316445744786359201454599278231027712,
                '[hidden email]'},
               {433883298582611803841718934712646521460354973696,
                '[hidden email]'},
               {456719261665907161938651510223838443642478919680,
                '[hidden email]'},
               {479555224749202520035584085735030365824602865664,
                '[hidden email]'},
               {502391187832497878132516661246222288006726811648,
                '[hidden email]'},
               {525227150915793236229449236757414210188850757632,
                '[hidden email]'},
               {548063113999088594326381812268606132370974703616,
                '[hidden email]'},
               {570899077082383952423314387779798054553098649600,
                '[hidden email]'},
               {593735040165679310520246963290989976735222595584,
                '[hidden email]'},
               {616571003248974668617179538802181898917346541568,
                '[hidden email]'},
               {639406966332270026714112114313373821099470487552,
                '[hidden email]'},
               {662242929415565384811044689824565743281594433536,
                '[hidden email]'},
               {685078892498860742907977265335757665463718379520,
                '[hidden email]'},
               {707914855582156101004909840846949587645842325504,
                '[hidden email]'},
               {730750818665451459101842416358141509827966271488,
                '[hidden email]'},
               {753586781748746817198774991869333432010090217472,
                '[hidden email]'},
               {776422744832042175295707567380525354192214163456,
                '[hidden email]'},
               {799258707915337533392640142891717276374338109440,
                '[hidden email]'},
               {822094670998632891489572718402909198556462055424,
                '[hidden email]'},
               {844930634081928249586505293914101120738586001408,
                '[hidden email]'},
               {867766597165223607683437869425293042920709947392,
                '[hidden email]'},
               {890602560248518965780370444936484965102833893376,
                '[hidden email]'},
               {913438523331814323877303020447676887284957839360,
                '[hidden email]'},
               {936274486415109681974235595958868809467081785344,
                '[hidden email]'},
               {959110449498405040071168171470060731649205731328,
                '[hidden email]'},
               {981946412581700398168100746981252653831329677312,
                '[hidden email]'},
               {1004782375664995756265033322492444576013453623296,
                '[hidden email]'},
               {1027618338748291114361965898003636498195577569280,
                '[hidden email]'},
               {1050454301831586472458898473514828420377701515264,
                '[hidden email]'},
               {1073290264914881830555831049026020342559825461248,
                '[hidden email]'},
               {1096126227998177188652763624537212264741949407232,
                '[hidden email]'},
               {1118962191081472546749696200048404186924073353216,
                '[hidden email]'},
               {1141798154164767904846628775559596109106197299200,
                '[hidden email]'},
               {1164634117248063262943561351070788031288321245184,
                '[hidden email]'},
               {1187470080331358621040493926581979953470445191168,
                '[hidden email]'},
               {1210306043414653979137426502093171875652569137152,
                '[hidden email]'},
               {1233142006497949337234359077604363797834693083136,
                '[hidden email]'},
               {1255977969581244695331291653115555720016817029120,
                '[hidden email]'},
               {1278813932664540053428224228626747642198940975104,
                '[hidden email]'},
               {1301649895747835411525156804137939564381064921088,
                '[hidden email]'},
               {1324485858831130769622089379649131486563188867072,
                '[hidden email]'},
               {1347321821914426127719021955160323408745312813056,
                '[hidden email]'},
               {1370157784997721485815954530671515330927436759040,
                '[hidden email]'},
               {1392993748081016843912887106182707253109560705024,
                '[hidden email]'},
               {1415829711164312202009819681693899175291684651008,
                '[hidden email]'},
               {1438665674247607560106752257205091097473808596992,
                '[hidden email]'}]},
             {dict,0,16,16,8,80,48,
                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}}ok
([hidden email])2>


Zubair


On Wed, Feb 10, 2010 at 5:48 PM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Two more questions:

- Is this on a single-node or multi-node cluster?
- Can you get to the Riak console (start Riak with "./bin/riak console", run "io:format("~p", [riak_ring_manager:get_my_ring()])." and send me the output?

Finally, to clarify my instructions on deleting the "data" directory, you will need to shut down Riak, then rename or delete the ./rel/riak/data directory, and then restart Riak.

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:43 AM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Hmmm... that is strange. Can you provide the following information to help me troubleshoot?:

- Can you post the actual error message that you see?
- Are you using the same backend on each system? Which backend are you using?
- Approximately how many buckets have you created, and how many objects are stored in total?
- If you delete or rename the ./rel/riak/data directory and write more objects, can you get the timeout to repeat? If so, when do you start to see the problem?

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:30 AM, Zubair Quraishi <[hidden email]> wrote:
Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put objects into Riak fine on both platforms, but when I do list_buckets( ) it doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone else experienced this?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Rusty Klophaus
Hi Zubair,

Yes, the current implementation of list_buckets expects all nodes to be present, otherwise it has no way of finding all of the buckets, so this is caused by 192.168.1.5 being down. 

We've thought about different options, such as adding a boolean flag to list_buckets that tells the system whether to proceed on error or timeout. 

The upshot is that, list_buckets (just like list_keys) is not generally something you want to do often, as it is resource intensive when you have a large number of objects.

Best,
Rusty

On Wed, Feb 10, 2010 at 12:02 PM, Zubair Quraishi <[hidden email]> wrote:
Yes, of course. Some extra info I forgot earlier is that I have two nodes, 192.168.1.3 and 192.168.1.5, and 192.168.1.5 is down.

- I'm using the default backend

- just 1 bucket I think (groceries :)

-  {ok, C} = riak:client_connect('[hidden email]').
{ok,{riak_client,'[hidden email]',<<0,38,76,114>>}}
([hidden email])2> C:list_buckets().

=ERROR REPORT==== 10-Feb-2010::17:49:58 ===
** State machine <5066.1052.0> terminating
** Last event in was timeout
** When State == waiting_kl
**      Data  == {state,<0.46.0>,plain,
                     {sbf,3.0e-5,0.85,1,0,
                         [{bloom,4.500000000000001e-6,11784498,24,0,
                              [{array,621379,0,0,1000000},
                               {array,621379,0,0,1000000},
                               {array,621379,0,0,1000000},
...
                               {array,621379,0,0,1000000},
                               {array,621379,0,0,1000000},
                               {array,621379,0,0,1000000}]}]},
                     [{45671926166590716193865151022383844364247891968,
                       '[hidden email]'},
                      {68507889249886074290797726533575766546371837952,
                       '[hidden email]'},
                      {91343852333181432387730302044767688728495783936,
                       '[hidden email]'},
                      {114179815416476790484662877555959610910619729920,
                       '[hidden email]'},
                      {137015778499772148581595453067151533092743675904,
                       '[hidden email]'},
                      {159851741583067506678528028578343455274867621888,
                       '[hidden email]'},
                      {182687704666362864775460604089535377456991567872,

...
                      {1050454301831586472458898473514828420377701515264,

                       '[hidden email]'},
                      {1347321821914426127719021955160323408745312813056,
                       '[hidden email]'},
                      {1370157784997721485815954530671515330927436759040,
                       '[hidden email]'},
                      {1392993748081016843912887106182707253109560705024,
                       '[hidden email]'},
                      {1415829711164312202009819681693899175291684651008,
                       '[hidden email]'},
                      {1438665674247607560106752257205091097473808596992,
                       '[hidden email]'}],
                     '_',120000,112621262,
                     {chstate,'[hidden email]',
                         [{'[hidden email]',{32,63432664825}}],
                         {64,
                          [{0,'[hidden email]'},
                           {22835963083295358096932575511191922182123945984,
                            '[hidden email]'},
                           {45671926166590716193865151022383844364247891968,
                            '[hidden email]'},
                           {68507889249886074290797726533575766546371837952,
                            '[hidden email]'},
                           {91343852333181432387730302044767688728495783936,
                            '[hidden email]'},
                           {114179815416476790484662877555959610910619729920,
                            '[hidden email]'},
                           {137015778499772148581595453067151533092743675904,
                            '[hidden email]'},
                           {159851741583067506678528028578343455274867621888,
                            '[hidden email]'},
...
                            '[hidden email]'},
                           {1415829711164312202009819681693899175291684651008,
                            '[hidden email]'},
                           {1438665674247607560106752257205091097473808596992,
                            '[hidden email]'}]},
                         {dict,0,16,16,8,80,48,
                             {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                             {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                               []}}}}}
** Reason for termination =
** {function_clause,
       [{riak_keys_fsm,waiting_kl,
            [timeout,
             {state,<0.46.0>,plain,
                 {sbf,3.0e-5,0.85,1,0,
                     [{bloom,4.500000000000001e-6,11784498,24,0,
                          [{array,621379,0,0,1000000},
                           {array,621379,0,0,1000000},
                           {array,621379,0,0,1000000},
...
                           {array,621379,0,0,1000000},
                           {array,621379,0,0,1000000}]}]},
                 [{45671926166590716193865151022383844364247891968,
                   '[hidden email]'},
                  {68507889249886074290797726533575766546371837952,
                   '[hidden email]'},
...           
                  {1233142006497949337234359077604363797834693083136,
                   '[hidden email]'},
                  {1255977969581244695331291653115555720016817029120,
                   '[hidden email]'},
                  {1278813932664540053428224228626747642198940975104,
                   '[hidden email]'},
                  {1438665674247607560106752257205091097473808596992,
                   '[hidden email]'}],
                 '_',120000,112621262,
                 {chstate,'[hidden email]',
                     [{'[hidden email]',{32,63432664825}}],
                     {64,
                      [{0,'[hidden email]'},
                       {22835963083295358096932575511191922182123945984,
                        '[hidden email]'},
.......
                        '[hidden email]'},
                       {1392993748081016843912887106182707253109560705024,
                        '[hidden email]'},
                       {1415829711164312202009819681693899175291684651008,
                        '[hidden email]'},
                       {1438665674247607560106752257205091097473808596992,
                        '[hidden email]'}]},
                     {dict,0,16,16,8,80,48,
                         {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                         {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],
                           []}}}}}]},
        {gen_fsm,handle_msg,7},
        {proc_lib,init_p_do_apply,3}]}
** at node [hidden email] **


- Yes, deleting data/ring did work, so I guess when the second Riak node dropped it caused some problem?

- multi-node, but one node down

- users-Mac-Pro:riak root# ./bin/riak console
Exec: /opt/local/bin/riak/rel/riak/erts-5.7.4/bin/erlexec -boot /opt/local/bin/riak/rel/riak/releases/0.8/riak -embedded -config /opt/local/bin/riak/rel/riak/etc/app.config -args_file /opt/local/bin/riak/rel/riak/etc/vm.args -- console
Root: /opt/local/bin/riak/rel/riak
Erlang R13B03 (erts-5.7.4) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:5] [hipe] [kernel-poll:true]

** Bad path can't read /opt/local/bin/riak/rel/riak/lib/kernel-2.13.4.ez/kernel-2.13.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/stdlib-1.16.4.ez/stdlib-1.16.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/sasl-2.1.8.ez/sasl-2.1.8/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/crypto-1.6.3.ez/crypto-1.6.3/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/runtime_tools-1.8.2.ez/runtime_tools-1.8.2/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/mochiweb-0.01.ez/mochiweb-0.01/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/webmachine-1.5.2.ez/webmachine-1.5.2/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/riak-0.7.1.ez/riak-0.7.1/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/xmerl-1.2.3.ez/xmerl-1.2.3/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/webtool-0.8.5.ez/webtool-0.8.5/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/tools-2.6.5.ez/tools-2.6.5/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/syntax_tools-1.6.4.ez/syntax_tools-1.6.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/ssl-3.10.7.ez/ssl-3.10.7/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/snmp-4.15.ez/snmp-4.15/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/public_key-0.4.ez/public_key-0.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/protobuffs-4.ez/protobuffs-4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/otp_mibs-1.0.6.ez/otp_mibs-1.0.6/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/os_mon-2.2.4.ez/os_mon-2.2.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/observer-0.9.8.1.ez/observer-0.9.8.1/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/mnesia-4.4.12.ez/mnesia-4.4.12/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/inets-5.2.ez/inets-5.2/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/hipe-3.7.4.ez/hipe-3.7.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/gs-1.5.11.ez/gs-1.5.11/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/eunit-2.1.4.ez/eunit-2.1.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/et-1.3.3.ez/et-1.3.3/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/erts-5.7.4.ez/erts-5.7.4/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/edoc-0.7.6.5.ez/edoc-0.7.6.5/ebin
** Bad path can't read /opt/local/bin/riak/rel/riak/lib/compiler-4.6.4.ez/compiler-4.6.4/ebin
** Found 0 name clashes in code paths
Eshell V5.7.4  (abort with ^G)
([hidden email])1>


([hidden email])1> io:format("~p", [riak_ring_manager:get_my_ring()]).
{ok,{chstate,'[hidden email]',[],

             {64,
              [{0,'[hidden email]'},
               {22835963083295358096932575511191922182123945984,
                '[hidden email]'},

               {45671926166590716193865151022383844364247891968,
                '[hidden email]'},
               {68507889249886074290797726533575766546371837952,
                '[hidden email]'},

               {91343852333181432387730302044767688728495783936,
                '[hidden email]'},
               {114179815416476790484662877555959610910619729920,
                '[hidden email]'},

               {137015778499772148581595453067151533092743675904,
                '[hidden email]'},
               {159851741583067506678528028578343455274867621888,
                '[hidden email]'},

               {182687704666362864775460604089535377456991567872,
                '[hidden email]'},
               {205523667749658222872393179600727299639115513856,
                '[hidden email]'},

               {228359630832953580969325755111919221821239459840,
                '[hidden email]'},
               {251195593916248939066258330623111144003363405824,
                '[hidden email]'},

               {274031556999544297163190906134303066185487351808,
                '[hidden email]'},
               {296867520082839655260123481645494988367611297792,
                '[hidden email]'},

               {319703483166135013357056057156686910549735243776,
                '[hidden email]'},
               {342539446249430371453988632667878832731859189760,
                '[hidden email]'},

               {365375409332725729550921208179070754913983135744,
                '[hidden email]'},
               {388211372416021087647853783690262677096107081728,
                '[hidden email]'},

               {411047335499316445744786359201454599278231027712,
                '[hidden email]'},
               {433883298582611803841718934712646521460354973696,
                '[hidden email]'},
               {456719261665907161938651510223838443642478919680,
                '[hidden email]'},
               {479555224749202520035584085735030365824602865664,
                '[hidden email]'},
               {502391187832497878132516661246222288006726811648,
                '[hidden email]'},
               {525227150915793236229449236757414210188850757632,
                '[hidden email]'},
               {548063113999088594326381812268606132370974703616,
                '[hidden email]'},
               {570899077082383952423314387779798054553098649600,
                '[hidden email]'},
               {593735040165679310520246963290989976735222595584,
                '[hidden email]'},
               {616571003248974668617179538802181898917346541568,
                '[hidden email]'},
               {639406966332270026714112114313373821099470487552,
                '[hidden email]'},
               {662242929415565384811044689824565743281594433536,
                '[hidden email]'},
               {685078892498860742907977265335757665463718379520,
                '[hidden email]'},
               {707914855582156101004909840846949587645842325504,
                '[hidden email]'},
               {730750818665451459101842416358141509827966271488,
                '[hidden email]'},
               {753586781748746817198774991869333432010090217472,
                '[hidden email]'},
               {776422744832042175295707567380525354192214163456,
                '[hidden email]'},
               {799258707915337533392640142891717276374338109440,
                '[hidden email]'},
               {822094670998632891489572718402909198556462055424,
                '[hidden email]'},
               {844930634081928249586505293914101120738586001408,
                '[hidden email]'},
               {867766597165223607683437869425293042920709947392,
                '[hidden email]'},
               {890602560248518965780370444936484965102833893376,
                '[hidden email]'},
               {913438523331814323877303020447676887284957839360,
                '[hidden email]'},
               {936274486415109681974235595958868809467081785344,
                '[hidden email]'},
               {959110449498405040071168171470060731649205731328,
                '[hidden email]'},
               {981946412581700398168100746981252653831329677312,
                '[hidden email]'},
               {1004782375664995756265033322492444576013453623296,
                '[hidden email]'},
               {1027618338748291114361965898003636498195577569280,
                '[hidden email]'},

               {1050454301831586472458898473514828420377701515264,
                '[hidden email]'},
               {1073290264914881830555831049026020342559825461248,
                '[hidden email]'},
               {1096126227998177188652763624537212264741949407232,
                '[hidden email]'},
               {1118962191081472546749696200048404186924073353216,
                '[hidden email]'},
               {1141798154164767904846628775559596109106197299200,
                '[hidden email]'},
               {1164634117248063262943561351070788031288321245184,
                '[hidden email]'},
               {1187470080331358621040493926581979953470445191168,
                '[hidden email]'},
               {1210306043414653979137426502093171875652569137152,
                '[hidden email]'},

               {1233142006497949337234359077604363797834693083136,
                '[hidden email]'},
               {1255977969581244695331291653115555720016817029120,
                '[hidden email]'},

               {1278813932664540053428224228626747642198940975104,
                '[hidden email]'},
               {1301649895747835411525156804137939564381064921088,
                '[hidden email]'},
               {1324485858831130769622089379649131486563188867072,

                '[hidden email]'},
               {1347321821914426127719021955160323408745312813056,
                '[hidden email]'},

               {1370157784997721485815954530671515330927436759040,
                '[hidden email]'},
               {1392993748081016843912887106182707253109560705024,
                '[hidden email]'},

               {1415829711164312202009819681693899175291684651008,
                '[hidden email]'},
               {1438665674247607560106752257205091097473808596992,
                '[hidden email]'}]},

             {dict,0,16,16,8,80,48,
                   {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
                   {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}}}ok
([hidden email])2>


Zubair


On Wed, Feb 10, 2010 at 5:48 PM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Two more questions:

- Is this on a single-node or multi-node cluster?
- Can you get to the Riak console (start Riak with "./bin/riak console", run "io:format("~p", [riak_ring_manager:get_my_ring()])." and send me the output?

Finally, to clarify my instructions on deleting the "data" directory, you will need to shut down Riak, then rename or delete the ./rel/riak/data directory, and then restart Riak.

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:43 AM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Hmmm... that is strange. Can you provide the following information to help me troubleshoot?:

- Can you post the actual error message that you see?
- Are you using the same backend on each system? Which backend are you using?
- Approximately how many buckets have you created, and how many objects are stored in total?
- If you delete or rename the ./rel/riak/data directory and write more objects, can you get the timeout to repeat? If so, when do you start to see the problem?

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:30 AM, Zubair Quraishi <[hidden email]> wrote:
Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put objects into Riak fine on both platforms, but when I do list_buckets( ) it doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone else experienced this?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com





_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Zubair Quraishi
In reply to this post by Rusty Klophaus
If I have a 100 node cluster and I did a "list_buckets" would it time out?

On Wed, Feb 10, 2010 at 5:48 PM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Two more questions:

- Is this on a single-node or multi-node cluster?
- Can you get to the Riak console (start Riak with "./bin/riak console", run "io:format("~p", [riak_ring_manager:get_my_ring()])." and send me the output?

Finally, to clarify my instructions on deleting the "data" directory, you will need to shut down Riak, then rename or delete the ./rel/riak/data directory, and then restart Riak.

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:43 AM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Hmmm... that is strange. Can you provide the following information to help me troubleshoot?:

- Can you post the actual error message that you see?
- Are you using the same backend on each system? Which backend are you using?
- Approximately how many buckets have you created, and how many objects are stored in total?
- If you delete or rename the ./rel/riak/data directory and write more objects, can you get the timeout to repeat? If so, when do you start to see the problem?

Thanks,
Rusty


On Wed, Feb 10, 2010 at 11:30 AM, Zubair Quraishi <[hidden email]> wrote:
Hi, I am using Riak 0.7.1 on both Mac OS X and Ubuntu 8 and 9. I can put objects into Riak fine on both platforms, but when I do list_buckets( ) it doesn't work on Mac OS X (it times out). It works fine on Ubuntu. Has anyone else experienced this?
_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com




_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Zubair Quraishi
In reply to this post by Rusty Klophaus
Hi Rusty,

How long would it take to implement list_buckets still showing the buckets when a node goes down? I disagree with you where you say list_buckets is not used that often, as I think that list buckets is a common operation for DBAs, and if it stops working when Riak goes down then it should be removed altogether, as a dynamically scalable database shouldn't stop working when a machine goes down (I thought this was the whole point of being fault tolerant). Anyway, since I am trying to make a tool to browse a Riak cluster how else can I get a list of the buckets, which is a common operation in my use case of a Riak database browser.

Thanks

Zubair

On Wed, Feb 10, 2010 at 6:08 PM, Rusty Klophaus <[hidden email]> wrote:
Hi Zubair,

Yes, the current implementation of list_buckets expects all nodes to be present, otherwise it has no way of finding all of the buckets, so this is caused by 192.168.1.5 being down. 

We've thought about different options, such as adding a boolean flag to list_buckets that tells the system whether to proceed on error or timeout. 

The upshot is that, list_buckets (just like list_keys) is not generally something you want to do often, as it is resource intensive when you have a large number of objects.

Best,
Rusty



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

bryan-basho
Administrator
On Thu, Feb 11, 2010 at 5:44 AM, Zubair Quraishi <[hidden email]> wrote:
> How long would it take to implement list_buckets still showing the buckets
> when a node goes down?


It depends on what our answer is, ultimately.  There are many options
for how it should be done, and each has its own effect on the
guarantees that Riak would provide about that data.  It's something
we're looking into.


> I disagree with you where you say list_buckets is not
> used that often, as I think that list buckets is a common operation for
> DBAs, and if it stops working when Riak goes down then it should be removed
> altogether, as a dynamically scalable database shouldn't stop working when a
> machine goes down (I thought this was the whole point of being fault
> tolerant).


Rusty's phrasing was unclear.  You may want to list_buckets, but you
shouldn't do it often in a production environment, with Riak's current
implementation.  We disagree about removing it altogether right now,
because it is a very useful operation, especially when you're in
development (non-production) mode, often dealing with half-complete
sets of test data, etc.  We recognize that there is some level of
demand for a production-ready version of this operation, and we are
evaluating our options for implementing one.


> Anyway, since I am trying to make a tool to browse a Riak cluster
> how else can I get a list of the buckets, which is a common operation in my
> use case of a Riak database browser.


We've implemented quite a few in-house Riak browsers over the last
couple of years.  Many of them weren't used on production datasets, so
listing buckets was fine.  The ones that were used in production
either knew of a pre-defined list of buckets to check, or asked the
user to type in a bucket to examine.  Since buckets, in a sense,
"always exist" (the list_buckets operation just lists buckets that
currently have data in them), asking the user for a bucket to examine
isn't really that awkward of an idea.

-Bryan

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Riak 0.7.1 list_buckets times out on Mac OS X but not on Ubuntu

Zubair Quraishi
Hi guys,

 Ok, thanks, that was quite a complete answer! :) I guess the more I think about it the more I see you are right. I was thinking there would only be a few easily cachable bucket names (<100) on most systems but after reading more in the Riak user archives I realise that some people may want to store individual records in their own bucket, thereby creating potentials billions of buckets, and not so easy to cache. Anyway, I think I can probably do something like just cache the bucket names in a specially named bucket together with a reference count for the number of items in each bucket for now, and just do a list_keys on that bucket, as a temporary solution for now. Anyway, thanks for the in depth explanation.

Zubair


> How long would it take to implement list_buckets still showing the buckets
> when a node goes down?


It depends on what our answer is, ultimately.  There are many options
for how it should be done, and each has its own effect on the
guarantees that Riak would provide about that data.  It's something
we're looking into.


> I disagree with you where you say list_buckets is not
> used that often, as I think that list buckets is a common operation for
> DBAs, and if it stops working when Riak goes down then it should be removed
> altogether, as a dynamically scalable database shouldn't stop working when a
> machine goes down (I thought this was the whole point of being fault
> tolerant).


Rusty's phrasing was unclear.  You may want to list_buckets, but you
shouldn't do it often in a production environment, with Riak's current
implementation.  We disagree about removing it altogether right now,
because it is a very useful operation, especially when you're in
development (non-production) mode, often dealing with half-complete
sets of test data, etc.  We recognize that there is some level of
demand for a production-ready version of this operation, and we are
evaluating our options for implementing one.


> Anyway, since I am trying to make a tool to browse a Riak cluster
> how else can I get a list of the buckets, which is a common operation in my
> use case of a Riak database browser.


We've implemented quite a few in-house Riak browsers over the last
couple of years.  Many of them weren't used on production datasets, so
listing buckets was fine.  The ones that were used in production
either knew of a pre-defined list of buckets to check, or asked the
user to type in a bucket to examine.  Since buckets, in a sense,
"always exist" (the list_buckets operation just lists buckets that
currently have data in them), asking the user for a bucket to examine
isn't really that awkward of an idea.

-Bryan


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com