HAProxy community

Can't access HAProxy externally from Internet


#1

I’m having an issue (timeout) trying to access the load balancer outside of my LAN (externally from the internet).
I can access it just fine from within my local network.

Some of the isolation tests I’ve tried:

  • My network firewall has both 80 and 443 ports open
    • Port forward them to HAProxy server and I can locally access https://example.com
    • Port forward them to HAProxy server and I get TIMEOUT outside of the network trying to access https://example.com
    • Port forward directly to example http server and I can locally access https://example.com
    • Port forward directly to example http server and I can access https://example.com outside of the network
    • This proves that traffic from the outside can hit a machine on the local network
  • Ran wget http://localhost:80 on the HAProxy server and received OK 200
    • Ran same command (with LB internal IP) from another server on same private network and got OK 200
    • Ran same command (with LB external domain http://example.com) from another server on same private network and got OK 200
    • This proves that LB is listening on 80
  • Ran wget --no-check-certificate https://localhost:443 on the HAProxy server and received OK 200
    • Ran same command (with LB internal IP) from another server on same private network and got OK 200
    • Ran same command (with LB external domain https://example.com) from another server on same private network and got OK 200
    • This proves that LB is listening on port 443

This leads me to believe there is something wrong with my version of HAProxy or my config.
Why would everything work fine internally but then timeout when a request is coming from outside the network?

$ haproxy -vv

HA-Proxy version 1.9.4-1ppa1~xenial 2019/02/07 - https://haproxy.org/
Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.21 2016-01-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE
              h2 : mode=HTTP       side=FE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

Current Config

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen haproxy-monitoring
        bind *:1000
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /
        stats auth XXX:XXX

frontend http_frontend
        bind *:80
        http-request redirect scheme https code 301 if !{ ssl_fc }

frontend https_frontend
        bind *:443 ssl crt /etc/ssl/private/
        mode http
        option httplog
        default_backend host_lamp

backend host_lamp
        mode http
        balance roundrobin
        server lamp1 X.X.X.X:80

Failing while trying to access from another ISP (My 4G cell hotspot)

* Preparing request to https://example.com/
* Using libcurl/7.57.0-DEV OpenSSL/1.0.2o zlib/1.2.11 libssh2/1.7.0_DEV
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 6 cookies
* Connection 75 seems to be dead!
* Closing connection 75
* TLSv1.2 (OUT), TLS alert, Client hello (1):
*   Trying X.X.X.X...
* TCP_NODELAY set
* connect to X.X.X.X port 443 failed: Timed out
* Failed to connect to example.com port 443: Timed out
* Closing connection 76

Success while trying to access from the same network

* Preparing request to https://example.com/
* Using libcurl/7.57.0-DEV OpenSSL/1.0.2o zlib/1.2.11 libssh2/1.7.0_DEV
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 6 cookies
* Hostname in DNS cache was stale, zapped
*   Trying X.X.X.X...
* TCP_NODELAY set
* Connected to example.com (X.X.X.X) port 443 (#77)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:\Users\XXX\AppData\Local\Temp\insomnia_6.3.2\2017-09-20.pem
*   CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=example.com
*  start date: Jan 31 23:00:29 2019 GMT
*  expire date: May  1 23:00:29 2019 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: example.com
> User-Agent: insomnia/6.3.2
> Accept: */*
< HTTP/1.1 200 OK
< Date: Wed, 27 Feb 2019 03:55:10 GMT
< Server: Apache
< X-Powered-By: PHP/7.1.25
< X-Frame-Options: SAMEORIGIN
< X-Mod-Pagespeed: 1.13.35.2-0
< Vary: Accept-Encoding
< Cache-Control: max-age=0, no-cache, s-maxage=10
< Content-Length: 70
< Content-Type: text/html; charset=UTF-8

* Received 70 B chunk
* Connection #77 to host example.com left intact

#2

Check your local iptables rules. Tcpdump it.


#3

Hmm I don’t have iptables or ufw or tcpdump. What other apps are available on ubuntu 16 to manage ports?

I noticed that IPv6 isn’t open for http and https. My 4g hotspot “What’s my ip” resolves to an IPv6 address. My local network’s public IP is IPv4. Maybe this is the answer?

> netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      187/haproxy
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      187/haproxy
tcp        0      0 0.0.0.0:1936            0.0.0.0:*               LISTEN      187/haproxy
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      178/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      178/sshd

#4

I don’t know what that means. If you lack the userspace applications, install them.


#5

Was the netstat cmd response not enough?

I trashed the Ubuntu image I was using and tried again on another image. Used the same config from above.and everything worked.

Thanks!


#6

No, it’s not enough. Haproxy is working perfectly fine and listening on the correct socket as per the netstat output. You have some problem unrelated to haproxy and the socket, which is why you need iptables tooling.


#7

<$0.02>
Some thoughts:
Ubuntu 18.04 is LTS, you should consider using it.
HAProxy is in Ubuntu’s repos, updated on newer releases (apt-get)
I feel like you may be hosting httpd & HAProxy on the same OS, but may be wrong - only one program can be bound to port 80/tcp.
Any internal firewall is, imho, a strict requirement for any public server - iptables is great, if you know how to use it. UFW is way simpler and much easier to learn.

There’s also HATop, which can be had throught apt-get. It’s a bit clunky, but may help with diagnostics. I’m thinking maybe thats what you’re using on port 1000, but not sure.

</$0.02>


#8

Thanks for the input AeSix.
I will switch to 18 as soon as possible.
Nothing additional installed onto the Ubuntu 16 LXC image, just HAProxy.
It’s a bare bones image, that’s why I asked if there was any other basic tooling that would manage firewall rules. (Or automatically block traffic outside of the private network.)

The port 1000 was used for the webUI metrics in HAProxy.

The solve was to try HaProxy 1.9 on a much larger Ubuntu 16.04 KVM image. Then it worked as expected.
Thanks!