Fwd: Re: Re: too_many_results error

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Re: Re: too_many_results error

Eric Moritz

That sounds like a bug rather that a feature to me. For instance if I wrote 100,001 blog entries, the search engine breaks hard for a query trying to find my latest tweets.

Is that what would happen?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:42 AM
Subject: Re: Re: too_many_results error
To: "Eric Moritz" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>

Hi Eric,

This is a failsafe that is applied prior to the 'rows' parameter. 

It is applied separately to provide a hard system limit, intended to allow the cluster administrator to guard against a malicious user, a client application that accidentally requests too much data, etc.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:38 AM, Eric Moritz <[hidden email]> wrote:

Would using the rows URL parameter fix this or is the error occurring prior to the result being limited?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:34 AM
Subject: Re: too_many_results error
To: "Malka Feldman" <[hidden email]>
Cc: <[hidden email]>

Hi Malka,

The error you see is caused by a protective configuration setting that is new in the 0.14 release. By default, the system will stop the query after 100,000 results and return an error to the system. It is intended to prevent Riak Search from grabbing all available memory when faced with a large result set. 

You can override this by setting the 'max_search_results' setting in the etc/app.config file. 

Best,
Rusty

On Sun, Jan 16, 2011 at 7:11 AM, Malka Feldman <[hidden email]> wrote:
Hi 
I try to get results from query through solr , but I get the error of too many results
I know I can have hundreds of thousands of answers, can it hold them?
the error :
{error,
    {throw,
        {too_many_results,
            {scope,#Ref<0.0.78.140604>,"user_profile","value",
                {scope,#Ref<0.0.78.140603>,undefined,"country",
                    {term,#Ref<0.0.78.140609>,<<"USA">>,
                        [{'[hidden email]',28438},
                         {'[hidden email]',28438},
                         {'[hidden email]',28438}],
                        28438.0,1.0,
                        #Fun}}}},
        [{riak_search_client,'-search/5-fun-0-',4},
         {riak_search_client,fold_results,5},
         {riak_search_client,search,6},
         {riak_search_client,search_doc,6},
         {riak_solr_searcher_wm,run_query,1},
         {riak_solr_searcher_wm,to_xml,2},
         {webmachine_resource,resource_call,3},
         {webmachine_resource,do,3}]}}

thanks,
--

Malka Feldman
Tribase LTD.

41 Shimon Hatzadik St.

Elad, Israel

Tel. 074-7122736  Fax. 03-9075211 Cell. 972-54-8370828

[hidden email][hidden email]

www.3basegroup.com


cid:image001.jpg@01CB2DDB.3AC492C0



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: too_many_results error

Rusty Klophaus
That's one way of looking at it. The other viewpoint is that given the choice between returning a partial (a.k.a. incorrect) result set vs. failing loudly, the safest option is to fail loudly.

Ideally there would be a way to return a partial result set with an indication in Solr that the result set is partial, but:

1) The Solr spec doesn't indicate a way to include such a message (at least none that I could find)
2) This is the first iteration of this feature, so we decided to keep it simple until we have more detailed information on what people need.

So with that in mind, how would you prefer this be handled? I'm open to rethinking the behavior.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:53 AM, Eric Moritz <[hidden email]> wrote:

That sounds like a bug rather that a feature to me. For instance if I wrote 100,001 blog entries, the search engine breaks hard for a query trying to find my latest tweets.

Is that what would happen?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:42 AM
Subject: Re: Re: too_many_results error
To: "Eric Moritz" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>

Hi Eric,

This is a failsafe that is applied prior to the 'rows' parameter. 

It is applied separately to provide a hard system limit, intended to allow the cluster administrator to guard against a malicious user, a client application that accidentally requests too much data, etc.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:38 AM, Eric Moritz <[hidden email]> wrote:

Would using the rows URL parameter fix this or is the error occurring prior to the result being limited?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:34 AM
Subject: Re: too_many_results error
To: "Malka Feldman" <[hidden email]>
Cc: <[hidden email]>

Hi Malka,

The error you see is caused by a protective configuration setting that is new in the 0.14 release. By default, the system will stop the query after 100,000 results and return an error to the system. It is intended to prevent Riak Search from grabbing all available memory when faced with a large result set. 

You can override this by setting the 'max_search_results' setting in the etc/app.config file. 

Best,
Rusty

On Sun, Jan 16, 2011 at 7:11 AM, Malka Feldman <[hidden email]> wrote:
Hi 
I try to get results from query through solr , but I get the error of too many results
I know I can have hundreds of thousands of answers, can it hold them?
the error :
{error,
    {throw,
        {too_many_results,
            {scope,#Ref<0.0.78.140604>,"user_profile","value",
                {scope,#Ref<0.0.78.140603>,undefined,"country",
                    {term,#Ref<0.0.78.140609>,<<"USA">>,
                        [{'[hidden email]',28438},
                         {'[hidden email]',28438},
                         {'[hidden email]',28438}],
                        28438.0,1.0,
                        #Fun}}}},
        [{riak_search_client,'-search/5-fun-0-',4},
         {riak_search_client,fold_results,5},
         {riak_search_client,search,6},
         {riak_search_client,search_doc,6},
         {riak_solr_searcher_wm,run_query,1},
         {riak_solr_searcher_wm,to_xml,2},
         {webmachine_resource,resource_call,3},
         {webmachine_resource,do,3}]}}

thanks,
--

Malka Feldman
Tribase LTD.

41 Shimon Hatzadik St.

Elad, Israel

Tel. 074-7122736  Fax. 03-9075211 Cell. 972-54-8370828

[hidden email][hidden email]

www.3basegroup.com


cid:image001.jpg@01CB2DDB.3AC492C0



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: too_many_results error

John Axel Eriksson
Well, I'm in this situation atm. I guess I could go and change the max_results setting but really - this actually
means that I can't search my data and to me, it's more important to be able to search than to get an error when
my index has grown too large. I understand that there's nothing for this in the solr spec, but getting an error because
I've indexed too much data is kind of unexpected.

I'm not familiar with the internals but I would've expected the rows parameter to be applied before the max_results limit - not
the other way around. I suspect there's a reason for this, but still - my data is now in an unsearchable state.

16 jan 2011 kl. 15.06 skrev Rusty Klophaus:

That's one way of looking at it. The other viewpoint is that given the choice between returning a partial (a.k.a. incorrect) result set vs. failing loudly, the safest option is to fail loudly.

Ideally there would be a way to return a partial result set with an indication in Solr that the result set is partial, but:

1) The Solr spec doesn't indicate a way to include such a message (at least none that I could find)
2) This is the first iteration of this feature, so we decided to keep it simple until we have more detailed information on what people need.

So with that in mind, how would you prefer this be handled? I'm open to rethinking the behavior.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:53 AM, Eric Moritz <[hidden email]> wrote:

That sounds like a bug rather that a feature to me. For instance if I wrote 100,001 blog entries, the search engine breaks hard for a query trying to find my latest tweets.

Is that what would happen?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:42 AM
Subject: Re: Re: too_many_results error
To: "Eric Moritz" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>

Hi Eric,

This is a failsafe that is applied prior to the 'rows' parameter. 

It is applied separately to provide a hard system limit, intended to allow the cluster administrator to guard against a malicious user, a client application that accidentally requests too much data, etc.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:38 AM, Eric Moritz <[hidden email]> wrote:

Would using the rows URL parameter fix this or is the error occurring prior to the result being limited?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:34 AM
Subject: Re: too_many_results error
To: "Malka Feldman" <[hidden email]>
Cc: <[hidden email]>

Hi Malka,

The error you see is caused by a protective configuration setting that is new in the 0.14 release. By default, the system will stop the query after 100,000 results and return an error to the system. It is intended to prevent Riak Search from grabbing all available memory when faced with a large result set. 

You can override this by setting the 'max_search_results' setting in the etc/app.config file. 

Best,
Rusty

On Sun, Jan 16, 2011 at 7:11 AM, Malka Feldman <[hidden email]> wrote:
Hi 
I try to get results from query through solr , but I get the error of too many results
I know I can have hundreds of thousands of answers, can it hold them?
the error :
{error,
    {throw,
        {too_many_results,
            {scope,#Ref<0.0.78.140604>,"user_profile","value",
                {scope,#Ref<0.0.78.140603>,undefined,"country",
                    {term,#Ref<0.0.78.140609>,<<"USA">>,
                        [{'[hidden email]',28438},
                         {'[hidden email]',28438},
                         {'[hidden email]',28438}],
                        28438.0,1.0,
                        #Fun}}}},
        [{riak_search_client,'-search/5-fun-0-',4},
         {riak_search_client,fold_results,5},
         {riak_search_client,search,6},
         {riak_search_client,search_doc,6},
         {riak_solr_searcher_wm,run_query,1},
         {riak_solr_searcher_wm,to_xml,2},
         {webmachine_resource,resource_call,3},
         {webmachine_resource,do,3}]}}

thanks,
--

Malka Feldman
Tribase LTD.
41 Shimon Hatzadik St.

Elad, Israel
Tel. 074-7122736  Fax. 03-9075211 Cell. 972-54-8370828

cid:image001.jpg@01CB2DDB.3AC492C0



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: too_many_results error

Rusty Klophaus
Hi John,

It sounds like there might be a misunderstanding here. The error is not indicating that the index has grown to large, it's indicating that you are running a query with many, many results - enough to potentially gobble up all of your memory. Rather than let this happen, Riak Search tries to detect the situation and fails the query rather than crashing. 

The intention is that most people will never bump up against the max_results setting. The few who do will have special use cases with especially large queries, and when this happens, it's going to require just a little testing and tuning to find a number that is large enough to handle their queries while still small enough to prevent memory exhaustion. It sounds like you fall into this situation. If you are running on big servers, you can safely bump up the number, the value depends on both your available memory and your object size.

Best,
Rusty



On Thu, Mar 3, 2011 at 6:42 AM, John Axel Eriksson <[hidden email]> wrote:
Well, I'm in this situation atm. I guess I could go and change the max_results setting but really - this actually
means that I can't search my data and to me, it's more important to be able to search than to get an error when
my index has grown too large. I understand that there's nothing for this in the solr spec, but getting an error because
I've indexed too much data is kind of unexpected.

I'm not familiar with the internals but I would've expected the rows parameter to be applied before the max_results limit - not
the other way around. I suspect there's a reason for this, but still - my data is now in an unsearchable state.

16 jan 2011 kl. 15.06 skrev Rusty Klophaus:

That's one way of looking at it. The other viewpoint is that given the choice between returning a partial (a.k.a. incorrect) result set vs. failing loudly, the safest option is to fail loudly.

Ideally there would be a way to return a partial result set with an indication in Solr that the result set is partial, but:

1) The Solr spec doesn't indicate a way to include such a message (at least none that I could find)
2) This is the first iteration of this feature, so we decided to keep it simple until we have more detailed information on what people need.

So with that in mind, how would you prefer this be handled? I'm open to rethinking the behavior.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:53 AM, Eric Moritz <[hidden email]> wrote:

That sounds like a bug rather that a feature to me. For instance if I wrote 100,001 blog entries, the search engine breaks hard for a query trying to find my latest tweets.

Is that what would happen?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:42 AM
Subject: Re: Re: too_many_results error
To: "Eric Moritz" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>

Hi Eric,

This is a failsafe that is applied prior to the 'rows' parameter. 

It is applied separately to provide a hard system limit, intended to allow the cluster administrator to guard against a malicious user, a client application that accidentally requests too much data, etc.

Best,
Rusty


On Sun, Jan 16, 2011 at 8:38 AM, Eric Moritz <[hidden email]> wrote:

Would using the rows URL parameter fix this or is the error occurring prior to the result being limited?

---------- Forwarded message ----------
From: "Rusty Klophaus" <[hidden email]>
Date: Jan 16, 2011 8:34 AM
Subject: Re: too_many_results error
To: "Malka Feldman" <[hidden email]>
Cc: <[hidden email]>

Hi Malka,

The error you see is caused by a protective configuration setting that is new in the 0.14 release. By default, the system will stop the query after 100,000 results and return an error to the system. It is intended to prevent Riak Search from grabbing all available memory when faced with a large result set. 

You can override this by setting the 'max_search_results' setting in the etc/app.config file. 

Best,
Rusty

On Sun, Jan 16, 2011 at 7:11 AM, Malka Feldman <[hidden email]> wrote:
Hi 
I try to get results from query through solr , but I get the error of too many results
I know I can have hundreds of thousands of answers, can it hold them?
the error :
{error,
    {throw,
        {too_many_results,
            {scope,#Ref<0.0.78.140604>,"user_profile","value",
                {scope,#Ref<0.0.78.140603>,undefined,"country",
                    {term,#Ref<0.0.78.140609>,<<"USA">>,
                        [{'[hidden email]',28438},
                         {'[hidden email]',28438},
                         {'[hidden email]',28438}],
                        28438.0,1.0,
                        #Fun}}}},
        [{riak_search_client,'-search/5-fun-0-',4},
         {riak_search_client,fold_results,5},
         {riak_search_client,search,6},
         {riak_search_client,search_doc,6},
         {riak_solr_searcher_wm,run_query,1},
         {riak_solr_searcher_wm,to_xml,2},
         {webmachine_resource,resource_call,3},
         {webmachine_resource,do,3}]}}

thanks,
--

Malka Feldman
Tribase LTD.
41 Shimon Hatzadik St.

Elad, Israel
Tel. 074-7122736  Fax. 03-9075211 Cell. 972-54-8370828

cid:image001.jpg@01CB2DDB.3AC492C0



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com



_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com