I’m having an issue (timeout) trying to access the load balancer outside of my LAN (externally from the internet).
I can access it just fine from within my local network.
Some of the isolation tests I’ve tried:
- My network firewall has both 80 and 443 ports open
- Port forward them to HAProxy server and I can locally access https://example.com
- Port forward them to HAProxy server and I get TIMEOUT outside of the network trying to access https://example.com
- Port forward directly to example http server and I can locally access https://example.com
- Port forward directly to example http server and I can access https://example.com outside of the network
- This proves that traffic from the outside can hit a machine on the local network
- Ran
wget http://localhost:80on the HAProxy server and received OK 200- Ran same command (with LB internal IP) from another server on same private network and got OK 200
- Ran same command (with LB external domain http://example.com) from another server on same private network and got OK 200
- This proves that LB is listening on 80
- Ran
wget --no-check-certificate https://localhost:443on the HAProxy server and received OK 200- Ran same command (with LB internal IP) from another server on same private network and got OK 200
- Ran same command (with LB external domain https://example.com) from another server on same private network and got OK 200
- This proves that LB is listening on port 443
This leads me to believe there is something wrong with my version of HAProxy or my config.
Why would everything work fine internally but then timeout when a request is coming from outside the network?
$ haproxy -vv
HA-Proxy version 1.9.4-1ppa1~xenial 2019/02/07 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.21 2016-01-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Current Config
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen haproxy-monitoring
bind *:1000
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth XXX:XXX
frontend http_frontend
bind *:80
http-request redirect scheme https code 301 if !{ ssl_fc }
frontend https_frontend
bind *:443 ssl crt /etc/ssl/private/
mode http
option httplog
default_backend host_lamp
backend host_lamp
mode http
balance roundrobin
server lamp1 X.X.X.X:80
Failing while trying to access from another ISP (My 4G cell hotspot)
* Preparing request to https://example.com/
* Using libcurl/7.57.0-DEV OpenSSL/1.0.2o zlib/1.2.11 libssh2/1.7.0_DEV
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 6 cookies
* Connection 75 seems to be dead!
* Closing connection 75
* TLSv1.2 (OUT), TLS alert, Client hello (1):
* Trying X.X.X.X...
* TCP_NODELAY set
* connect to X.X.X.X port 443 failed: Timed out
* Failed to connect to example.com port 443: Timed out
* Closing connection 76
Success while trying to access from the same network
* Preparing request to https://example.com/
* Using libcurl/7.57.0-DEV OpenSSL/1.0.2o zlib/1.2.11 libssh2/1.7.0_DEV
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 6 cookies
* Hostname in DNS cache was stale, zapped
* Trying X.X.X.X...
* TCP_NODELAY set
* Connected to example.com (X.X.X.X) port 443 (#77)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: C:\Users\XXX\AppData\Local\Temp\insomnia_6.3.2\2017-09-20.pem
* CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=example.com
* start date: Jan 31 23:00:29 2019 GMT
* expire date: May 1 23:00:29 2019 GMT
* subjectAltName: host "example.com" matched cert's "example.com"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: example.com
> User-Agent: insomnia/6.3.2
> Accept: */*
< HTTP/1.1 200 OK
< Date: Wed, 27 Feb 2019 03:55:10 GMT
< Server: Apache
< X-Powered-By: PHP/7.1.25
< X-Frame-Options: SAMEORIGIN
< X-Mod-Pagespeed: 1.13.35.2-0
< Vary: Accept-Encoding
< Cache-Control: max-age=0, no-cache, s-maxage=10
< Content-Length: 70
< Content-Type: text/html; charset=UTF-8
* Received 70 B chunk
* Connection #77 to host example.com left intact