Forwarding SSL traffic through HAProxy with a self signed cert fails, works when pointed directly at the backend (no haproxy)

I have a couple other tests i want to run, but have tried what i thought should work with the verify none. FWIW, this is a staging environment emulating a production environment, which is set up on a bunch of cloud servers. The staging environment is an Ubuntu box running a bunch of LCX containers.

In staging - I have created a CA, and built on that a self signed certificate. In the configuration, below, one F/E HTTPS server that points at currently two B/E HTTPS Nginx servers. I am not interested in terminating SSL as i don’t do that in production as i would need to establish SSL again due to the cloud servers being on a semi public network.

If i point curl, or openssl s_client -connect directly at the nginx servers www.staging.example.org i can see the certificates and with browsers through untrusted i can use the site normally. When i go through HAProxy with curl -k I see
curl: (35) gnutls_handshake() failed: The TLS connection was non-properly terminated.

With openssl s_client i see `CONNECTED(00000003)
140350987986584:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 305 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1471959614
Timeout : 300 (sec)
Verify return code: 0 (ok)
—`

With FireFox/Chrome there are slight different messages, but both result in connection refused. With any of these modes changing my host file to a direct nginx server i get the proper data, but the expected warning that the connection is not secure due to the self-signed cert. Enough babble, but hopefully sufficient information for assistance, here is the relevant portion of the haproxy.cfg, which is nearly identical to produciton (which works wonderfully, backed by a Lets’s Encrypt CA)

`global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-server-verify none
ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
ssl-default-bind-options no-sslv3

defaults
log global
option log-health-checks
mode tcp
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000

errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend http
mode tcp
bind 0.0.0.0:80
default_backend http_backends

backend http_backends
mode tcp
option tcp-check
balance roundrobin
default-server inter 1s fall 2

server stagingweb02 10.1.10.98:80 check send-proxy
server stagingweb01 10.1.10.191:80 check send-proxy

frontend https
#redirect scheme https if !{ ssl_fc }
mode tcp
bind 0.0.0.0:443
option tcplog
timeout http-request 10s
default_backend https_backends

backend https_backends
mode tcp
balance roundrobin
default-server inter 1s fall 2

server stagingweb02 10.1.10.98:443 send-proxy ssl verify none
server stagingweb01 10.1.10.191:443 send-proxy ssl verify none`

FWIW, i’ve tried a couple configurations, but currently the .pem files are concatenated with both the CRT and the CA. I’ve tried it a couple different ways and have added my CA to /etc/ssl/certs with update-ca-certificates, since i am on Ubuntu 16.04, running HAProxy 1.6.8

In Summary, all posts point me to ssl verify none, should do the trick, but i’m missing something else, which will likely be obvious when seen.

You want haproxy to be a TCP proxy. Transferring TCP payload from the frontend to the backend. Then your haproxy configuration must not contain any ssl related configuration.

But what you told haproxy to do is to encrypt the TCP payload (which is actually SSL) once again on the backend.

Remove the ssl keyword from the server’s in the backend section and it will work. But I suggest you remove everything ssl related from this configuration, including verify and the ssl defaults in the global section, so that you don’t get confused when looking at this configuration.

Thank you @lukastribus for responding, seems like a logical move, but unfortunately it didn’t change the responses, mainly counting on curl, but similar in browsers (which caches responses). Here is my entire current config.

Chrome reports SSL_PROTOCOL_ERROR
FireFox shows: SSL_ERROR_RX_RECORD_TOO_LONG
Curl reports handshake error

Just in case i stopped the service for a handful of seconds and restarted. The requests don’t seem to be getting to the backends at all as there is nothing in the nginx logs.

Again, thank you for taking the time to look at my question and respond!!

cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

defaults
log global
option log-health-checks
mode tcp
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000

errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend http
mode tcp
bind 0.0.0.0:80
default_backend http_backends

backend http_backends
mode tcp
option tcp-check
balance roundrobin
default-server inter 1s fall 2

server stagingweb02 10.1.10.98:80 check send-proxy
server stagingweb01 10.1.10.191:80 check send-proxy

frontend https
mode tcp
bind 0.0.0.0:443
option tcplog
timeout http-request 10s
default_backend https_backends

backend https_backends
mode tcp
balance roundrobin
default-server inter 1s fall 2

server stagingweb02 10.1.10.98:443 check send-proxy
server stagingweb01 10.1.10.191:443 check send-proxy

frontend api
mode tcp
bind 0.0.0.0:8000
option tcplog
timeout http-request 20s
default_backend api_backends

backend api_backends
mode tcp
default-server inter 1s fall 2

server stagingweb02 10.1.10.98:8000 check send-proxy
server stagingweb01 10.1.10.191:8000 check send-proxy

frontend hastats
mode http
bind 0.0.0.0:5134
default_backend hastats_be

backend hastats_be
mode http
log global
stats enable
stats hide-version
stats realm "My Stats"
stats uri /hap?dastats
stats auth xxx:xxxxxxx

listen galera
bind 0.0.0.0:3306
balance roundrobin
mode tcp
option tcpka
option mysql-check user haproxy

server stagingdb02 10.1.10.76:3306 check weight 1
server stagingdb01 10.1.10.80:3306 check weight 1
server stagingdb03 10.1.10.82:3306 check weight 1

Are you positive your backend support the proxy-protocol? Is it support and enabled?

Please try without proxy-protocol to see if that is causing the issue.

its the same code as production, but hey its worth a try, removed from both nginx and haproxy

original nginx server block
listen 443 ssl http2 default_server proxy_protocol;
listen 10.1.10.191:443 ssl http2; // this was placed here to access the server directly - commenting out for this test

but alas this second line that was causing 1/2 of the issue, i removed all proxy_protocol from nginx and send-proxy from haproxy and it worked, so i put back all send-proxy and proxy_protocol in haproxy and nginx respectively and all was fine and i’m getting the correct client address.

Ultimately it was a combination of SSL options in HAProxy and attempt to bypass proxy_protocol with that second configuration line (the one with direct ip and no proxy_protocol), which was causing the issue. Thanks again for your assistance, very much appreciate the second pair of eyes as i had started at this for too long.

Best,
Jeff