HAProxy not switching between backends

ksl28 · December 14, 2017, 11:39am

Hi,

Im having an issue with HA Proxy, that its redirecting all traffic to the default backend.
In an previous config, HA Proxy managed to redirect just fine - so the hostname is defined in the TCP package.

Ive tried several different settings, but cant seem to find a solution that works.

Config:

global
#log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 2m
maxconn 80000
user haproxy
group haproxy
daemon

defaults
mode http
log global
#option httplog
option dontlognull
timeout client 300s

Statistics

listen stats
bind 192.168.62.148:8040
mode http
stats enable
stats uri /lbstats
stats realm Please\ Login
stats auth admin:admin
stats admin if TRUE

###################################################################################################

FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND

###################################################################################################

frontend FT_HTTPS_IN
mode tcp
bind 192.168.62.148:443
tcp-request content accept if { req_ssl_hello_type 1 }
use_backend BCK_RDS_HTTPS if { req_ssl_sni -i remote .domain .dk }
use_backend BCK_RDS_GW_HTTPS if { req_ssl_sni -i rdsgw .domain .dk }
use_backend BCK_OWNCLOUD_HTTPS if { req_ssl_sni -i oc .domain .dk }
use_backend BCK_POC_WEB if { req_ssl_sni -i poc .domain .dk }
option tcp-smart-accept
default_backend BCK_Exchange_HTTPS

###################################################################################################

BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND

###################################################################################################

backend BCK_Exchange_HTTPS
mode tcp
retries 3
timeout server 300s
timeout connect 10s
server S1EXCH01 192.168.60.82:443 check #ssl verify none

backend BCK_RDS_GW_HTTPS
mode tcp
retries 3
timeout server 300s
timeout connect 10s
balance roundrobin
server S1TSGW03 192.168.62.90:443 check #ssl verify none
server S1TSGW04 192.168.62.92:443 check #ssl verify none

backend BCK_RDS_HTTPS
mode tcp
retries 3
timeout server 300s
timeout connect 10s
server S1TSWEB01 192.168.62.93:443 check #ssl verify none

backend BCK_OWNCLOUD_HTTPS
mode tcp
retries 3
timeout server 300s
timeout connect 10s
server S1OC03 192.168.62.100:443 check #ssl verify none

backend BCK_POC_WEB
mode tcp
retries 3
timeout server 300s
timeout connect 10s
server S1POC01 192.168.62.91:443 check #ssl verify none

###################################################################################################

HA Proxy Config

###################################################################################################

The topic is short, since i cant seem to find any misconfigured settings.
PS: the domains are seperated by spaces, since new users can only post 2 links in a topic.

reinaldo.gomes · December 14, 2017, 1:32pm

Why is there a blank space before every dot on your domains?

ksl28 · December 14, 2017, 2:05pm

the domains are seperated by spaces, since new users can only post 2 links in a topic.
It isnt like that in the real config

lukastribus · December 14, 2017, 2:38pm

The inspect-delay is missing in the frontend:

tcp-request inspect-delay 5s

ksl28 · December 14, 2017, 3:46pm

Hi,

Thx for the reply.
Ive already fixed that, after i posted the config.
That didnt change the issue im afraid

lukastribus · December 14, 2017, 3:51pm

I can’t spot anything wrong with the configuration, it must be something else.

Does the certificate served by the backends have SAN’s or wildcard that overlap with each other (like a wildcard certificate of domain.dk or SANs that contain more than one hostname)?

Can you post the output of haproxy -vv please?

Does this happen with all clients?

ksl28 · December 14, 2017, 3:58pm

hi,

Thats somehow great to hear

Config:

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.2g-fips 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

All of the backend servers, do use the same wildcard certificate.
An Cisco ASA 5506 nats port 443 from a public IP to the HA Proxy.
The HA Proxy should then “forward” the sessions, to the correct backend.

From what i have been able to test - yes (Mobile 4G, Laptop, External Server)

lukastribus · December 14, 2017, 4:08pm

And that is the issue. If you want SNI routing you need exact certificates, you cannot have the same backend, because the browser will use existing SSL sessions to speak to all domains covered by the certificate, however the SNI routing only works with the client_hello, before the SSL handshake.

So what happens is that when you start the browser and request remote.domain.dk, it will work. But when you open rdsgw.domain.dk, you browser will reuse the existing SSL session, which is connected to the remote.domain.dk backend.

Either use non-overlapping certificates or terminate SSL on haproxy and use the Host header in the HTTP Protocol to select the correct backend. This won’t work with protocols other than HTTP though.

ksl28 · December 14, 2017, 4:20pm

Hi,

I never would have thought if that, since it has worked before with this backend infrastructure.

Since HA Proxy is only servicing HTTPS traffic, i guess that it could work.

What do i need to change in the config, to terminate the SSL traffic on the HA Proxy?

lukastribus · December 14, 2017, 4:30pm

disable HTTPS on the backends and enable port 80.
install the SSL certificate on the haproxy box and refer to it in the config
configure SSL properly
switch from “mode tcp” to “mode http” (you can inherit it from the defaults section)
content-switch based on the Host header (replace req_ssl_sni with “hdr(host)”)

So the configuration would probably look like this:

global
 #log 127.0.0.1 local2
 chroot /var/lib/haproxy
 pidfile /var/run/haproxy.pid
 stats socket /var/run/haproxy.sock mode 600 level admin
 stats timeout 2m
 maxconn 80000
 user haproxy
 group haproxy
 daemon
 tune.ssl.default-dh-param 2048
 ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
 ssl-default-bind-options no-sslv3 no-tls-tickets


defaults
 mode http
 log global
 #option httplog
 option dontlognull
 timeout client 300s

# Statistics

listen stats
 bind 192.168.62.148:8040
 mode http
 stats enable
 stats uri /lbstats
 stats realm Please\ Login
 stats auth admin:admin
 stats admin if TRUE

###################################################################################################
FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND
###################################################################################################

frontend FT_HTTPS_IN
 bind 192.168.62.148:80 ssl crt /path/to/wildcard.pem   
 use_backend BCK_RDS_HTTPS if { hdr(host) -i remote.domain.dk }
 use_backend BCK_RDS_GW_HTTPS if { hdr(host) -i rdsgw.domain.dk }
 use_backend BCK_OWNCLOUD_HTTPS if { hdr(host) -i oc.domain.dk }
 use_backend BCK_POC_WEB if { hdr(host) -i poc.domain.dk }
 option tcp-smart-accept
 default_backend BCK_Exchange_HTTPS

###################################################################################################
BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND
###################################################################################################

backend BCK_Exchange_HTTPS
 retries 3
 timeout server 300s
 timeout connect 10s
 server S1EXCH01 192.168.60.82:80 check #ssl verify none

backend BCK_RDS_GW_HTTPS
 retries 3
 timeout server 300s
 timeout connect 10s
 balance roundrobin
 server S1TSGW03 192.168.62.90:80 check #ssl verify none
 server S1TSGW04 192.168.62.92:80 check #ssl verify none

backend BCK_RDS_HTTPS
 retries 3
 timeout server 300s
 timeout connect 10s
 server S1TSWEB01 192.168.62.93:80 check #ssl verify none

backend BCK_OWNCLOUD_HTTPS
 retries 3
 timeout server 300s
 timeout connect 10s
 server S1OC03 192.168.62.100:80 check #ssl verify none

backend BCK_POC_WEB
 retries 3
 timeout server 300s
 timeout connect 10s
 server S1POC01 192.168.62.91:80 check #ssl verify none

ksl28 · December 14, 2017, 4:32pm

Hi,

Thx for the reply.
I dont want to have the HA Proxy controlling the certificate.
Isnt it possible for the proxy, to just route to a backend, based on the host-header - and ignore the certificate?

lukastribus · December 14, 2017, 4:33pm

No, because it’s encrypted.

ksl28 · December 14, 2017, 4:38pm

Isnt it only the package (data) that is encrypted? the meta data (host-header) isnt?

This have worked before with the same backend servers and certificate, so i believe it should work?

lukastribus · December 14, 2017, 4:47pm

No, HTTPS encrypts both headers and body. Otherwise secret cookies and sessions keys would be transmitted in plaintext, which would defeat the whole purpose of HTTPS.

If it worked before, did you actually confirm the theory:

close the browser completely
reopen it
access one of those hostnames, and it will access the correct backend
only subsequent request to a different backend would fail (hit the backend you tried previously)

ksl28 · December 14, 2017, 4:55pm

Hi,

That might have been the case…
There is a minor issue, that the backend servers dont allow http (80) - they need HTTPS.
Is it possible, to have the HA Proxy listen on port 443 with the certificate, and also have the backend servers running HTTPS?

lukastribus · December 14, 2017, 5:38pm

Yes, but you need to install the certificate with private key on haproxy.

To re-encrypt towards your backend, just use the ssl keyword:

server S1POC01 192.168.62.91:443 check ssl verify none

ksl28 · December 15, 2017, 1:01pm

Hi,

I will try this soon, and report back.
Thank you so much, for taking the time to help out

Topic		Replies	Views
Redirect not working properly to backends Help!	0	212	February 24, 2024
Need help with haproxy config Help!	0	558	March 20, 2019
Use_backend app-main if { hdr_dom(host) -i } -> Not recognize domain Help!	5	4272	November 7, 2019
Newbie - Several backends - not redirecting correctly Help!	7	1977	September 22, 2016
Haproxy use_backend condition (acl) is missing the backend Help!	2	1078	January 4, 2020

HAProxy not switching between backends

Statistics

FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND - FRONTEND

BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND - BACKEND

HA Proxy Config

Related topics