enotconn error

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

enotconn error

Travis Kirstine

Hi all,

 

I’m using haproxy with a riak 2.1.4 in a 5 node cluster.  I’m getting fairly consistent enotconn errors in riak which happen to coincide with client failures.  We’ve setup haproxy as recommended (https://gist.github.com/gburd/1507077) see below.  I’m running a leveldb backend with 9 GB max memory (I can go higher if needed).  I’m not sure at this point if I have a network issue or leveldb / riak issue.

 

2016-09-14 08:10:16 =CRASH REPORT====

  crasher:

    initial call: mochiweb_acceptor:init/3

    pid: <0.28104.442>

    registered_name: []

    exception error: {function_clause,[{webmachine_request,peer_from_peername,[{error,enotconn},{webmachine_request,{wm_reqstate,#Port<0.6539192>,[],undefined,undefined,undefined,{wm_reqdata,'GET',http,{1,0},"defined_in_wm_req_srv_init","defined_in_wm_req_srv_init",defined_on_call,defined_in_load_dispatch_data,"/ping","/ping",[],defined_in_load_dispatch_data,"defined_in_load_dispatch_data",500,1073741824,67108864,[],[],{0,nil},not_fetched_yet,false,{0,nil},<<>>,follow_request,undefined,undefined,[]},undefined,undefined,undefined}}],[{file,"src/webmachine_request.erl"},{line,150}]},{webmachine_request,get_peer,1,[{file,"src/webmachine_request.erl"},{line,124}]},{webmachine,new_request,2,[{file,"src/webmachine.erl"},{line,69}]},{webmachine_mochiweb,loop,2,[{file,"src/webmachine_mochiweb.erl"},{line,49}]},{mochiweb_http,headers,5,[{file,"src/mochiweb_http.erl"},{line,96}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

    ancestors: ['http://192.168.18.64:8098_mochiweb',riak_api_sup,<0.319.0>]

    messages: []

    links: [<0.325.0>,#Port<0.6539192>]

    dictionary: []

    trap_exit: false

    status: running

    heap_size: 987

    stack_size: 27

    reductions: 963

  neighbours:

 

 

# haproxy.cfg

global

    log         127.0.0.1 local2 info

    chroot      /var/lib/haproxy

    pidfile     /var/run/haproxy.pid

    maxconn     256000

    user        haproxy

    group       haproxy

    spread-checks       5

    daemon

    quiet

    stats socket /var/lib/haproxy/stats

defaults

    log                     global

    option                  httplog

    option                  dontlognull

    option                  redispatch

    timeout connect         5000

    maxconn                 256000

 

frontend  main *:80

    mode                    http

    acl url_static       path_beg       -i /static /images /javascript /stylesheets

    acl url_static       path_end       -i .jpg .gif .png .css .js

 

backend static

    balance     roundrobin

    server      static 127.0.0.1:4331 check

 

backend app

    mode                    http

    balance     roundrobin

    server  wmts1riak 192.168.18.72:80 check

    server  wmts2riak 192.168.18.73:80 check

 

backend riak_rest_backend

       mode               http

       balance            roundrobin

       option             httpchk GET /ping

       option             httplog

       server riak1 192.168.18.64:8098 weight 1 maxconn 1024  check

       server riak2 192.168.18.65:8098 weight 1 maxconn 1024  check

       server riak3 192.168.18.66:8098 weight 1 maxconn 1024  check

       server riak4 192.168.18.67:8098 weight 1 maxconn 1024  check

       server riak5 192.168.18.68:8098 weight 1 maxconn 1024  check

 

frontend riak_rest

       bind               *:8098

       mode               http

       option             contstats

       default_backend    riak_rest_backend

 

backend riak_protocol_buffer_backend

       balance            leastconn

       mode               tcp

       option             tcpka

       option             srvtcpka

       server riak1 192.168.18.64:8087 weight 1 maxconn 1024  check

       server riak2 192.168.18.65:8087 weight 1 maxconn 1024  check

       server riak3 192.168.18.66:8087 weight 1 maxconn 1024  check

       server riak4 192.168.18.67:8087 weight 1 maxconn 1024  check

       server riak5 192.168.18.68:8087 weight 1 maxconn 1024  check

 

frontend riak_protocol_buffer

       bind               *:8087

       mode               tcp

       option             tcplog

       option             contstats

       mode               tcp

       option             tcpka

       option             srvtcpka

       default_backend    riak_protocol_buffer_backend

 

 

 

 

 

 

 

 

 

 

 

 


_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reply | Threaded
Open this post in threaded view
|

Re: enotconn error

Luke Bakken
Travis -

What are the client failures you are seeing? What Riak client library
are you using, and are you using the PB or HTTP interface to Riak?

The error message you provided indicates that the ping request
returned from Riak after haproxy closed the socket for the request.
One cause would be very high server load causing timeouts.
--
Luke Bakken
Engineer
[hidden email]


On Wed, Sep 14, 2016 at 7:23 AM, Travis Kirstine
<[hidden email]> wrote:

> Hi all,
>
>
>
> I’m using haproxy with a riak 2.1.4 in a 5 node cluster.  I’m getting fairly
> consistent enotconn errors in riak which happen to coincide with client
> failures.  We’ve setup haproxy as recommended
> (https://gist.github.com/gburd/1507077) see below.  I’m running a leveldb
> backend with 9 GB max memory (I can go higher if needed).  I’m not sure at
> this point if I have a network issue or leveldb / riak issue.
>
>
>
> 2016-09-14 08:10:16 =CRASH REPORT====
>
>   crasher:
>
>     initial call: mochiweb_acceptor:init/3
>
>     pid: <0.28104.442>
>
>     registered_name: []
>
>     exception error:
> {function_clause,[{webmachine_request,peer_from_peername,[{error,enotconn},{webmachine_request,{wm_reqstate,#Port<0.6539192>,[],undefined,undefined,undefined,{wm_reqdata,'GET',http,{1,0},"defined_in_wm_req_srv_init","defined_in_wm_req_srv_init",defined_on_call,defined_in_load_dispatch_data,"/ping","/ping",[],defined_in_load_dispatch_data,"defined_in_load_dispatch_data",500,1073741824,67108864,[],[],{0,nil},not_fetched_yet,false,{0,nil},<<>>,follow_request,undefined,undefined,[]},undefined,undefined,undefined}}],[{file,"src/webmachine_request.erl"},{line,150}]},{webmachine_request,get_peer,1,[{file,"src/webmachine_request.erl"},{line,124}]},{webmachine,new_request,2,[{file,"src/webmachine.erl"},{line,69}]},{webmachine_mochiweb,loop,2,[{file,"src/webmachine_mochiweb.erl"},{line,49}]},{mochiweb_http,headers,5,[{file,"src/mochiweb_http.erl"},{line,96}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
>
>     ancestors: ['<a href="http://192.168.18.64:8098_mochiweb',riak_api_sup">http://192.168.18.64:8098_mochiweb',riak_api_sup,<0.319.0>]
>
>     messages: []
>
>     links: [<0.325.0>,#Port<0.6539192>]
>
>     dictionary: []
>
>     trap_exit: false
>
>     status: running
>
>     heap_size: 987
>
>     stack_size: 27
>
>     reductions: 963
>
>   neighbours:
>
>
>
>
>
> # haproxy.cfg
>
> global
>
>     log         127.0.0.1 local2 info
>
>     chroot      /var/lib/haproxy
>
>     pidfile     /var/run/haproxy.pid
>
>     maxconn     256000
>
>     user        haproxy
>
>     group       haproxy
>
>     spread-checks       5
>
>     daemon
>
>     quiet
>
>     stats socket /var/lib/haproxy/stats
>
> defaults
>
>     log                     global
>
>     option                  httplog
>
>     option                  dontlognull
>
>     option                  redispatch
>
>     timeout connect         5000
>
>     maxconn                 256000
>
>
>
> frontend  main *:80
>
>     mode                    http
>
>     acl url_static       path_beg       -i /static /images /javascript
> /stylesheets
>
>     acl url_static       path_end       -i .jpg .gif .png .css .js
>
>
>
> backend static
>
>     balance     roundrobin
>
>     server      static 127.0.0.1:4331 check
>
>
>
> backend app
>
>     mode                    http
>
>     balance     roundrobin
>
>     server  wmts1riak 192.168.18.72:80 check
>
>     server  wmts2riak 192.168.18.73:80 check
>
>
>
> backend riak_rest_backend
>
>        mode               http
>
>        balance            roundrobin
>
>        option             httpchk GET /ping
>
>        option             httplog
>
>        server riak1 192.168.18.64:8098 weight 1 maxconn 1024  check
>
>        server riak2 192.168.18.65:8098 weight 1 maxconn 1024  check
>
>        server riak3 192.168.18.66:8098 weight 1 maxconn 1024  check
>
>        server riak4 192.168.18.67:8098 weight 1 maxconn 1024  check
>
>        server riak5 192.168.18.68:8098 weight 1 maxconn 1024  check
>
>
>
> frontend riak_rest
>
>        bind               *:8098
>
>        mode               http
>
>        option             contstats
>
>        default_backend    riak_rest_backend
>
>
>
> backend riak_protocol_buffer_backend
>
>        balance            leastconn
>
>        mode               tcp
>
>        option             tcpka
>
>        option             srvtcpka
>
>        server riak1 192.168.18.64:8087 weight 1 maxconn 1024  check
>
>        server riak2 192.168.18.65:8087 weight 1 maxconn 1024  check
>
>        server riak3 192.168.18.66:8087 weight 1 maxconn 1024  check
>
>        server riak4 192.168.18.67:8087 weight 1 maxconn 1024  check
>
>        server riak5 192.168.18.68:8087 weight 1 maxconn 1024  check
>
>
>
> frontend riak_protocol_buffer
>
>        bind               *:8087
>
>        mode               tcp
>
>        option             tcplog
>
>        option             contstats
>
>        mode               tcp
>
>        option             tcpka
>
>        option             srvtcpka
>
>        default_backend    riak_protocol_buffer_backend
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> riak-users mailing list
> [hidden email]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>

_______________________________________________
riak-users mailing list
[hidden email]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com