Load balancing SSL with Pass through the traffic

Help!
1

Hi,

What I miss here? While redirecting from http to https, without entering the port number directly in the address, I am unable to connect to it.

For example, this doesn’t work:

https://consul.sre-test.sre.me/ui/dc1-test/nodes

while this one, does:

https://consul.sre-test.sre.me:8000/ui/dc1-test/nodes

haproxy.cfg:

frontend httpfront
  bind *:80
  mode http
  option httplog
  option forwardfor
  redirect scheme https code 301 if !{ ssl_fc }

frontend httpsfront
  bind *:443 ssl crt /data01/cert/haproxy.pem
  mode tcp
  option tcplog

  acl is_consul hdr_beg(host) -i consul.sre-test.sre.me
  
 use_backend consul  if is_consul

#---------------------------------------------------------------------
# round robin balancing between vault servers
#---------------------------------------------------------------------
backend consul
  balance roundrobin
  server is-srecore01t 10.53.132.76:8000 ssl verify required ca-file /data01/cert/ca2.pem
  server is-srecore02t 10.53.132.77:8000 ssl verify required ca-file /data01/cert/ca2.pem
  server is-srecore03t 10.53.132.78:8000 ssl verify required ca-file /data01/cert/ca2.pem

Any help on this question will be greatly appreciated.

Cheers,
BB

2

consul.sre-test.sre.me does not actually point to haproxy, but your backend? A firewall blocks port 443 and there is some equivalent service on port 8000?

Doesn’t work how? You get a timeout? You get a HTTP 500 error - from haproxy or your application?

If the hostname is consul.sre-test.sre.me but you are using hdr_beg to match it against consul.sre-test.sre.me, then it will never match, because it does not BEGIN with that. Use hdr, not hdr_beg for a exact match.

3

Hi,

Shame on me for not giving any background information.

That’s correct.

# dig consul.sre-test.sre.me
;; ANSWER SECTION:
consul.sre-test.sre.me. 2516 IN     A       10.53.132.76
consul.sre-test.sre.me. 2516 IN     A       10.53.132.78
consul.sre-test.sre.me. 2516 IN     A       10.53.132.77

Seems like no other applications are using those ports, 443 and 8000:

lsof -n -i4TCP:8000 | grep LISTEN
consul  2991 root   12u  IPv4 8899061      0t0  TCP 10.53.132.78:irdmi (LISTEN)

and

lsof -n -i4TCP:443 | grep LISTEN
haproxy 20929 root    8u  IPv4 9339041      0t0  TCP *:https (LISTEN)

All I got is ERR_EMPTY_RESPONSE in browser. In terminal, although, sometimes I’m receive HTTP/1.1 301 Moved Permanently

#curl -v --cert wild.sre-test.sre.me https://consul.sre-test.sre.me
* About to connect() to consul.sre-test.sre.met port 443 (#0)
*   Trying 10.53.132.78...
* Connected to consul.sre-test.sre.me (10.53.132.78) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=*.sre-test.sre.met,OU=TEST,OU=ETOOL,O=SBB,L=Melbourne,ST=Victoria,C=AU
*       start date: Dec 13 13:39:01 2019 GMT
*       expire date: Dec 12 13:39:01 2021 GMT
*       common name: *.sre-test.sre.me
*       issuer: CN=SBB Internal Issuing 2 CA 1,OU=PKI2,O=SBB,C=AU
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: consul.sre-test.sre.me
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< content-type: text/html; charset=utf-8
< location: /ui/
< date: Sat, 14 Dec 2019 22:18:24 GMT
< content-length: 39
<
<a href="/ui/">Moved Permanently</a>.

and sometimes curl: (52) Empty reply from server

# curl -v --cert wild.sre-test.sre.me https://consul.sre-test.sre.me
* About to connect() to consul.sre-test.sre.met port 443 (#0)
*   Trying 10.53.132.78...
* Connected to consul.sre-test.sre.me (10.53.132.78) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=*.sre-test.sre.met,OU=TEST,OU=ETOOL,O=SBB,L=Melbourne,ST=Victoria,C=AU
*       start date: Dec 13 13:39:01 2019 GMT
*       expire date: Dec 12 13:39:01 2021 GMT
*       common name: *.sre-test.sre.me
*       issuer: CN=SBB Internal Issuing 2 CA 1,OU=PKI2,O=SBB,C=AU
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: consul.sre-test.sre.me
> Accept: */*
>
* Empty reply from server
* Connection #0 to host consul.sre-test.oneadr.net left intact
curl: (52) Empty reply from server

Exchanging hdr_beg(host) for hdr(host) doesn’t change a thing.

A few lines from the log file:

Dec 14 22:54:58 is-srecore03t haproxy[20929]: 10.152.22.172:52522 [14/Dec/2019:22:54:58.282] httpfront httpfront/<NOSRV> 0/-1/-1/-1/0 301 112 - - LR-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
Dec 14 23:12:58 is-srecore03t haproxy[20929]: 10.53.132.78:54832 [14/Dec/2019:23:12:58.307] httpsfront~ consul/is-srecore03t 0/15/111 190 -- 1/1/0/0/0 0/0
Dec 14 23:18:16 is-srecore03t haproxy[20929]: 10.53.132.78:55030 [14/Dec/2019:23:18:16.835] httpsfront~ consul/is-srecore01t 0/5/100 190 -- 1/1/0/0/0 0/0
Dec 14 23:18:24 is-srecore03t haproxy[20929]: 10.53.132.78:55036 [14/Dec/2019:23:18:24.647] httpsfront~ consul/is-srecore02t 0/5/83 190 -- 1/1/0/0/0 0/0

And a little bit info about haproxy:

# haproxy -vv
HA-Proxy version 2.1.1 2019/12/11 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.1.html
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [CACHE] cache
        [FCGI] fcgi-app
        [TRACE] trace
        [COMP] compression
4

Your setup is completely unclear at this point.

Why would your DNS point to the multiple backend IPs while at the same time you are using that hostname, apparently to access haproxy.

Where does haproxy run?

5

I have three instances of haproxy running on three different hosts. Which one will serve depend on which one will be randomly chosen by DNS server. This is kind of HA.

6

In the meantime I made a side project using the same hardware architecture, logic and certificates but with nginx. It seems to be working fine so it suggest that something is wrong with soft configuration only.

7

After all night long reading I came to the conclusion that my above configuration has nothing to do with “SSL Pass Through” setup.
In addition to all mistake I made, it seems like normal ACL is not working for SSL but here ‘req_ssl_sni’ came to the rescue. And because there is no reason to make SSL termination for listener, afterall it’s “pass through”, I ended up using:

frontend http_frontend
    bind :80
    mode http
    redirect scheme https if !{ ssl_fc }

frontend https_frontend
    bind :443
    option tcplog
    mode tcp

    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

    use_backend consul if { req_ssl_sni -i consul.sre-test.oneadr.net }

backend consul
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server is-srecore01t 10.53.132.76:8000
    server is-srecore02t 10.53.132.77:8000
    server is-srecore03t 10.53.132.78:8000

Cheers,
BB

8

Sure, if there is no need for SSL termination, passing it through is way simpler.

That said, SSL termination and reencryption on the backend is perfectly supported and there are use-cases for it. I assume something failed between haproxy and your backend application. Maybe your backend required SNI and rejected the requests without SNI from Haproxy.