HAproxy needs to be restarted

KenynMacCormik · October 30, 2019, 9:48am

Greetings,

I’m using HAproxy for y Exchange servers. Outlook clients are working generally fine, but we have 3rd party application that sends a lot of emails. And here is a thing. Once a day this application looses the ability to communicate with Exchange servers through HAproxy. If I restart the Haproxy daemon it works fine until the very next day. I’m a bit lost and do not know where to start looking. Could you please anvice me on this matter?

Config

global
        log     127.0.0.1 local2 info
        chroot  /var/lib/haproxy
        pidfile /var/run/haproxy.pid
        maxconn 10000
        user    haproxy
        group   haproxy
        daemon

        ssl-default-bind-options no-sslv3
        tune.ssl.default-dh-param 2048
        ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

defaults
        log     global
        log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"

        #option tcplog
        option  redispatch
        option  http-keep-alive
        option  log-health-checks
        #option dontlognull
        #option dontlog-normal

        no option httpclose

        retries 3
        backlog 10000
        balance leastconn

        timeout connect         15s
        timeout http-keep-alive 30s
        timeout http-request    30s
        timeout queue           30s
        timeout tarpit          60s
        timeout check           10s
        timeout client          15m
        timeout server          15m

        default-server inter 5s rise 2 fall 3

frontend haproxy_stats_frt
        bind <private ip>.7:8080

        mode http

        acl is_internal src <private ip>.0/24

        stats   enable
        stats   refresh 10s
        stats   show-node
        stats   auth    admin:<password>
        stats   uri     /stats
        stats   admin if is_internal

frontend web_http_frt
        bind <private ip>.7:80

        mode    http

        option  forwardfor except 127.0.0.0/8
        option  httplog

        acl     is_haproxy      hdr(host)       -m str -i haproxy1.<private domain>
        acl     is_ca_web       hdr(host)       -m str -i ca.<private domain>
        acl     is_empty_query  query           -m found
        acl     is_ecp          path            -m reg -i (^\/ecp$|^\/ecp\/)
        acl     is_healthcheck  path            -m reg -i healthcheck\.htm$

        http-request deny if is_ecp
        http-request deny if is_healthcheck
        http-request redirect location https://%[hdr(host)]%[path] code 301 if !is_ca_web !is_empty_query !is_haproxy
        http-request redirect location https://%[hdr(host)]%[path]?%[query] code 301 if !is_ca_web is_empty_query !is_haproxy

        use_backend ca_web if is_ca_web

frontend web_https_frt
        bind <private ip>.7:443 ssl crt-list /etc/haproxy/cer.list

        mode    http

        option  forwardfor except 127.0.0.0/8
        option  httplog

        capture cookie SERVERID len 32

        capture request header Host             len 64
        capture request header User-agent       len 64
        capture request header X-Forwarded-For  len 64
        capture request header WWW-Authenticate len 64
        capture request header Authorization    len 64

        capture response header Location len 32
        capture response header Server len 32

        acl     is_ca_web               hdr(host)       -m str -i ca.<private domain>
        acl     is_autodiscover_host    hdr(host)       -m reg -i ^autodiscover.<private domain>$
        acl     is_mail                 hdr(host)       -m reg -i (^mail.<private domain>$|^mail1.<private domain>$)
        acl     is_empty_path           path            -m reg -i (^\/$|^$)
        acl     is_rpc                  path            -m reg -i (^\/rpc$|^\/rpc\/)
        acl     is_owa                  path            -m reg -i (^\/owa$|^\/owa\/)
        acl     is_ews                  path            -m reg -i (^\/ews$|^\/ews\/)
        acl     is_oab                  path            -m reg -i (^\/oab$|^\/oab\/)
        acl     is_eas                  path            -m reg -i (^\/eas$|^\/eas\/)
        acl     is_mapi                 path            -m reg -i (^\/mapi$|^\/mapi\/)
        acl     is_ecp                  path            -m reg -i (^\/ecp$|^\/ecp\/)
        acl     is_autodiscover_url     path            -m reg -i ^\/Autodiscover\/
        acl     is_activesync           path            -m reg -i (^\/Microsoft\-Server\-ActiveSync$|^\/Microsoft\-Server\-ActiveSync\/)
        acl     is_healthcheck          path            -m reg -i healthcheck.htm$

        http-request deny if is_ecp
        http-request deny if is_healthcheck
        http-request redirect location http://ca.<private domain>%[path] code 301 if is_ca_web

        use_backend     autodiscover_bck        if is_autodiscover_host
        use_backend     rpc_bck                 if is_rpc
        use_backend     owa_bck                 if is_owa
        use_backend     ews_bck                 if is_ews
        use_backend     oab_bck                 if is_oab
        use_backend     eas_bck                 if is_eas
        use_backend     mapi_bck                if is_mapi
        use_backend     owa_bck                 if is_mail is_empty_path
        use_backend     activesync_bck          if is_activesync

frontend mail_frt
        mode tcp

        option tcplog

        bind    <private ip>.7:25
        bind    <private ip>.7:110
        bind    <private ip>.7:143
        bind    <private ip>.7:587
        bind    <private ip>.7:993
        bind    <private ip>.7:995

        acl     is_pop          dst_port 110
        acl     is_imap         dst_port 143
        acl     is_smtps        dst_port 587
        acl     is_imaps        dst_port 993
        acl     is_pops         dst_port 995

        use_backend     smtps_bck       if is_smtps
        use_backend     imap_bck        if is_imap
        use_backend     pop_bck         if is_pop
        use_backend     imaps_bck       if is_imaps
        use_backend     pops_bck        if is_pops

        default_backend smtp_bck

backend smtps_bck
        mode tcp

        option  tcp-check

        tcp-check connect port 587
        tcp-check expect string 220

        server ex1 ex1.<private domain>:587 check
        server ex2 ex2.<private domain>:587 check

backend imap_bck
        mode tcp

        option  tcp-check

        tcp-check connect port 143
        tcp-check expect string * OK

        server ex1 ex1.<private domain>:143 check
        server ex2 ex2.<private domain>:143 check

backend pop_bck
        mode tcp

        option  tcp-check

        tcp-check connect port 110
        tcp-check expect string +OK

        server ex1 ex1.<private domain>:110 check
        server ex2 ex2.<private domain>:110 check

backend imaps_bck
        mode tcp

        option  tcp-check

        tcp-check connect port 993 ssl
        tcp-check expect string * OK

        server ex1 ex1.<private domain>:993 check verify none
        server ex2 ex2.<private domain>:993 check verify none

backend pops_bck
        mode tcp

        option  tcp-check

        tcp-check connect port 995 ssl
        tcp-check expect string +OK

        server ex1 ex1.<private domain>:995 check verify none
        server ex2 ex2.<private domain>:995 check verify none


backend smtp_bck
        mode tcp

        option  tcp-check

        tcp-check expect string 220

        server ex1 ex1.<private domain>:25 check
        server ex2 ex2.<private domain>:25 check

backend owa_bck
        mode http

        option httpchk GET /OWA/HealthCheck.htm
        http-check expect status 200

        server ex1 ex1.<private domain>:443 ssl check verify none
        server ex2 ex2.<private domain>:443 ssl check verify none

backend autodiscover_bck
        mode http

        option httpchk GET /autodiscover/HealthCheck.htm
        http-check expect status 200

        server ex1 ex1.<private domain>:443 ssl check verify none
        server ex2 ex2.<private domain>:443 ssl check verify none

backend rpc_bck
        mode http

        option httpchk GET /RPC/HealthCheck.htm
        http-check expect status 200
        cookie SERVERID insert nocache

        server ex1 ex1.<private domain>:443 ssl check verify none cookie rpc1
        server ex2 ex2.<private domain>:443 ssl check verify none cookie rpc2

backend ews_bck
        mode http

        option httpchk GET /EWS/HealthCheck.htm
        http-check expect status 200
        cookie SERVERID insert nocache

        server ex1 ex1.<private domain>:443 ssl check verify none cookie ews1
        server ex2 ex2.<private domain>:443 ssl check verify none cookie ews2

backend eas_bck
        mode http

        option httpchk GET /Microsoft-Server-ActiveSync/HealthCheck.htm
        http-check expect status 200

        server ex1 ex1.<private domain>:443 ssl check verify none
        server ex2 ex2.<private domain>:443 ssl check verify none

backend mapi_bck
        mode http

        option httpchk GET /mapi/HealthCheck.htm
        http-check expect status 200
        cookie SERVERID insert nocache

        server ex1 ex1.<private domain>:443 ssl check verify none id 103 cookie mapi1
        server ex2 ex2.<private domain>:443 ssl check verify none id 104 cookie mapi2

backend oab_bck
        mode http

        option httpchk GET /OAB/HealthCheck.htm
        http-check expect status 200

        server ex1 ex1.<private domain>:443 ssl check verify none
        server ex2 ex2.<private domain>:443 ssl check verify none

backend activesync_bck
        mode http

        option httpchk GET /Microsoft-Server-ActiveSync/HealthCheck.htm
        http-check expect status 200

        server ex1 ex1.<private domain>:443 ssl check verify none
        server ex2 ex2.<private domain>:443 ssl check verify none


backend ca_web
        mode http

        http-check expect string CRPT\ CA\ Info

        server web1 web1.<private domain>:80 check

jerome · October 30, 2019, 10:20am

what is your haproxy version ? if <2.0 you want to set maxconn in the defaults as well otherwise it defaults to 2000 for every frontend.
you want to look at haproxy logs to understand why haproxy fails until you restart it. Traffic logs can tell if you reach maxconn and haproxy starts queuing.

KenynMacCormik · October 30, 2019, 10:32am

Greetings,

I’ve added

maxconn 5000 to the defaults section. Could you please tell me which status code will tell me that connections starts queuing?

jerome · October 30, 2019, 10:36am

you want to look at the value for actconn or feconn in either http or tcp log format. tcp mode eats twice as many conns.

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.2.2
https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.2.3

KenynMacCormik · October 30, 2019, 10:42am

Thank you, I’ll wait till the problem arise again and see if there is queuing issue.

Topic		Replies	Views
Connection reset for LDAPS Help!	2	530	June 6, 2023
Randomly get 502 errors Help!	1	1084	November 28, 2022
HAProxy SSL Offloading high CPU usage on CentOS 7 Help!	0	743	June 13, 2017
HAProxy SSL Offloading high CPU usage on CentOS 7 Help!	2	2557	June 20, 2017
HAProxy fails on restart Help!	11	11581	February 17, 2019

HAproxy needs to be restarted

Related Topics