Clients from a specific WAN network can't access application layer


#1

Hi, I’m new to HAPROXY and I have a problem with some clients that cannot access haproxy services from a specific network on WAN.

When they try to access it from a web browser it keeps waiting for a response. The first thing I figured is routing, but ping works, and when I try to access the stats page y pops out the authentication window, but then it just keeps working and nothing is returned. If I try to direct access the service backends bypassing the haproxy it works.

This is my setting:

  • Servers: 2 CentOS 7 with keepalived cluster for HA.
  • HAProxy: 1.5.18 (the one that comes with this CentOS version via yum).
  • Network: 2 interfaces: ens160 (DMZ) and ens192 (LAN and WAN).
  • Routes: default gateway configured for ens160. Specific static routes added for ens192.
  • Firewall: ports 80, 443, 9000 and others are opened.
  • SELinux: enabled (I tried disabling it, but no luck).

HAProxy Configuration (I modified some sensible info):

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2 info

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    tune.ssl.default-dh-param 2048

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
    balance                 leastconn
    default-server          inter 3s rise 2 fall 3

#---------------------------------------------------------------------
# Exchange HTTPS Frontend
#---------------------------------------------------------------------
frontend exchange_https_frontend
    bind *:443 name https ssl crt /etc/ssl/certs/mycert.pem
    mode http
    option http-keep-alive
    no option httpclose
    no option http-server-close
    no option forceclose
    option contstats
    option dontlognull
    log global
    option httplog
    option forwardfor       except 127.0.0.0/8
    timeout client 25s
    timeout http-keep-alive 1s
    timeout http-request 15s
    maxconn 10000
    acl ssl_connection ssl_fc
    acl host_mail hdr(Host) -i my.mail.com
    acl path_slash path /
    acl path_owa path_beg -i /owa/
    acl path_ecp path_beg -i /ecp/
    acl path_ews path_beg -i /ews/
    acl path_activesync path_beg -i /Microsoft-Server-ActiveSync
    acl path_oa path_beg -i /rpc/rpcproxy.dll
    acl path_autodiscover path_beg -i /Autodiscover/Autodiscover.xml
    acl path_ps path_beg -i /Powershell/
    acl path_oab path_beg -i /oab/
    ###acl path_mapi path_beg -i /mapi/
    acl path_check path_end -i HealthCheck.htm
    # HTTP deny rules
    http-request deny if path_check
    # HTTP redirect rules
    http-request redirect scheme https code 302 unless ssl_connection
    http-request redirect location /owa/ code 302 if path_slash host_mail
    # HTTP routing rules
    use_backend exchange_https_owa_backend if path_owa
    use_backend exchange_https_ecp_backend if path_ecp
    use_backend exchange_https_ews_backend if path_ews
    use_backend exchange_https_activesync_backend if path_activesync
    use_backend exchange_https_oa_backend if path_oa
    use_backend exchange_https_autodiscover_backend if path_autodiscover
    use_backend exchange_https_ps_backend if path_ps
    use_backend exchange_https_oab_backend if path_oab
    ###use_backend exchange_https_mapi if path_mapi
    # other services go here
    default_backend exchange_https_default_backend

#---------------------------------------------------------------------
# Exchange HTTPS Backends
#---------------------------------------------------------------------

# Outlook Web Access (OWA):
backend exchange_https_owa_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /owa/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Exchange Control Panel (ECP):
backend exchange_https_ecp_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /ECP/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Exchange Web Services (EWS):
backend exchange_https_ews_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /EWS/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Exchange Active Sync (EAS):
backend exchange_https_activesync_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /Microsoft-Server-ActiveSync/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Outlook Anywhere (OA):
backend exchange_https_oa_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /RPC/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Autodiscover (AU):
backend exchange_https_autodiscover_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /Autodiscover/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# PowerShell (PS):
backend exchange_https_ps_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Offline Address Book (OAB):
backend exchange_https_oab_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    option httpchk GET /OAB/HealthCheck.htm
    http-check expect string 200\ OK
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check

# Exchange Default Backend:
backend exchange_https_default_backend
    option http-keep-alive
    option prefer-last-server
    no option httpclose
    no option http-server-close
    no option forceclose
    no option http-tunnel
    option forwardfor
    server server151 192.168.5.151:443 maxconn 10000 weight 10 ssl verify none check
    server server152 192.168.5.152:443 maxconn 10000 weight 10 ssl verify none check
    server server153 192.168.5.153:443 maxconn 10000 weight 10 ssl verify none check
    server server154 192.168.5.154:443 maxconn 10000 weight 10 ssl verify none check


#---------------------------------------------------------------------
# Exchange SMTP Settings
#---------------------------------------------------------------------
listen smtp25 *:25
    mode tcp
    option tcplog
    balance leastconn
    option tcp-check
    tcp-check expect string 220
    default-server inter 3s rise 2 fall 3
    server server151 192.168.5.151:25 check
    server server152 192.168.5.152:25 check
    server server153 192.168.5.153:25 check
    server server154 192.168.5.154:25 check

listen smtp587 *:587
    mode tcp
    option tcplog
    balance leastconn
    option tcp-check
    tcp-check expect string 220
    default-server inter 3s rise 2 fall 3
    server server151 192.168.5.151:587 check
    server server152 192.168.5.152:587 check
    server server153 192.168.5.153:587 check
    server server154 192.168.5.154:587 check

#---------------------------------------------------------------------
# HAProxy stats page:
#---------------------------------------------------------------------
listen stats :9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats uri /
    stats auth admin:mypassword

#2

Sounds like routing indeed. Routing fail for specific source IP’s or even specific destination or source ports, when network load-balances on broken paths based on the 5-tuple.

I’d suggest you tcpdump’ it based on the source IP and check what exactly happens on the wire.


#3

Thank you very much lukastribus!

Sadly I have poor network debuging skills… I try to tcpdump it and this is what I get when I try to access the haproxy stats page:

[root@myhaproxy ~]# tcpdump port 9000 -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
13:39:08.674573 IP 192.168.2.10.49510 > myhaproxy.cslistener: Flags [S], seq 4010027171, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:39:08.674625 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [S.], seq 2138253499, ack 4010027172, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:39:08.675026 IP 192.168.2.10.49511 > myhaproxy.cslistener: Flags [S], seq 3904051396, win 65535, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:39:08.675059 IP myhaproxy.cslistener > 192.168.2.10.49511: Flags [S.], seq 2279332806, ack 3904051397, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:39:08.703168 IP 192.168.2.10.49510 > myhaproxy.cslistener: Flags [.], ack 1, win 1024, length 0
13:39:08.705298 IP 192.168.2.10.49510 > myhaproxy.cslistener: Flags [P.], seq 1:348, ack 1, win 1024, length 347
13:39:08.705485 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:2921, ack 348, win 237, length 2920
13:39:08.705507 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 2921:5841, ack 348, win 237, length 2920
13:39:08.705538 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 5841:7301, ack 348, win 237, length 1460
13:39:08.705569 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 7301:7342, ack 348, win 237, length 41
13:39:08.705629 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 7342:10262, ack 348, win 237, length 2920
13:39:08.705644 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 10262:13182, ack 348, win 237, length 2920
13:39:08.705708 IP 192.168.2.10.49511 > myhaproxy.cslistener: Flags [.], ack 1, win 1024, length 0
13:39:08.726801 IP 192.168.2.10.49510 > myhaproxy.cslistener: Flags [.], ack 1, win 1024, options [nop,nop,sack 1 {7301:7342}], length 0
13:39:08.726828 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:08.954969 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:09.414038 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:10.333003 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:12.169012 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:15.844997 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:18.705153 IP myhaproxy.cslistener > 192.168.2.10.49511: Flags [F.], seq 1:213, ack 1, win 229, length 212
13:39:18.725324 IP 192.168.2.10.49511 > myhaproxy.cslistener: Flags [.], ack 214, win 1023, length 0
13:39:23.189049 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:39:37.876995 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460
13:40:07.220992 IP myhaproxy.cslistener > 192.168.2.10.49510: Flags [.], seq 1:1461, ack 348, win 237, length 1460

Regards


#4

If ping works it’s highly likely an asymmetric routing issue. The reason I say that is that ICMP is a connectionless/stateless or whatever protocol which is why it will work when you have such problems while TCP needs to see the whole connection at any stateful devices en the route or risk having the connection dropped.

Consider the routing hops client to HAproxy and HAproxy back to a client, they will need to go through the same devices the same way in both directions. If at any point it goes via a different router/firewall(Or maybe just a differen context) check your logs at that router for “Out of state” type errors to confirm.

Fixing it wil be a case of making the routing symmetric again.

That or you have it blocked by a firewall somewhere…


#5

Thanks Aaron, I think too that this is something in the routing setting. But the wierd thing is that other WAN locations work fine, and the routing settings are consistents.

I only have the default gateway configured over the interface ens160. All WAN machines are routed through the interface ens192, and those routes are statically added. All WAN machines works, except for some from this specific location.

Regards


#6

You need to look at a specific non-working client, and in the tcpdump, filter its IP. Then you check whether you actually receive anything from it, or not.

In this case, all traffic from and to 192.168.1.5 on source or destination port 9000 is captured in the file capture-traffic-haproxy.cap:
tcpdump -pns0 -w capture-traffic-haproxy.cap -i ens192 host 192.168.1.5 and port 9000


#7

After a while working with the tools that you recommended, we discovered a configuration error in the WAN TELCO’s router: the MTU size was wrong. After they changed it to the proper value, it worked fine.
Thank you all very much for your time and support.

Regards,
Enzo