HAProxy 1.8.14 and ADFS 4.0

Help!
1

Hi

I’m trying to get ADFS to work in HAProxy, and it works in simple TCP setup:

defaults
log global
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

frontend ADFSFrontend
bind 10.0.0.100:443
mode tcp
default_backend ADFSBackend

backend ADFSBackend
mode tcp
balance roundrobin
server 450adfs01 10.0.0.101:443 check
server 450adfs02 10.0.0.102:443 check

On the other hand I can get sni to work… but I can’t get any checks to work here - any ideas?

frontend ADFSFrontend
bind 10.0.0.100:443 ssl crt /etc/ssl/mycert.pem
mode tcp
default_backend ADFSBackend

backend ADFSBackend
mode tcp
balance roundrobin
option httpchk GET /adfs/ls/IdpInitiatedSignon.aspx HTTP/1.0\r\n
http-check expect string Sign\ in
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
server 450adfs01 10.0.0.101:443 ssl verify none sni ssl_fc_sni inter 3s rise 2 fall 3
server 450adfs02 10.0.0.102:443 ssl verify none sni ssl_fc_sni inter 3s rise 2 fall 3

Any comments are welcome

Best regards
Flemming

2

I assume you also need to set SNI for the health checks, and you can do that with check-sni

3

Hi Lukas

Yes, I beleive so… but just can’t get it to work

Med venlig hilsen/With best regards
Flemming Hougaard

SuperviseIT

4

What’s failing exactly, do you have the log messages?

5

Hi Lukas

My problem is that I have no knowledge how to use check-sni :frowning:

I’ve tried to use:

server 450adfs01 10.0.0.101:443 check-sni ssl_fc_sni inter 3s rise 2 fall 3 and similar

No errors, ADFS just stop working.

/F

Lukas Tribus via HAProxy community haproxy@discoursemail.com 05-03-19 0:09 >>>


lukastribus
March 4
What’s failing exactly, do you have the log messages?

6

You need to use the hostname that your backend server expects. You cannot use a fetch like ssl_fc_sni because there is no associated frontend connection where SNI could be extracted from.

server 450adfs01 10.0.0.101:443 ssl verify none check-sni myadfs.example.com sni ssl_fc_sni inter 3s rise 2 fall 3
7

Hi Lukas

I’ve tried that also, and it works… but in the webinterface under status, it’s listed as “No Check” - and that’s what I want to achieve, to enable checks.

/Flemming

Lukas Tribus via HAProxy community haproxy@discoursemail.com 5. marts 2019 23:50 >>>


lukastribus

    March 5

You need to use the hostname that your backend server expects. You cannot use a fetch like ssl_fc_sni because there is no associated frontend connection where SNI could be extracted from.

server 450adfs01 10.0.0.101:443 ssl verify none check-sni myadfs.example.com sni ssl_fc_sni inter 3s rise 2 fall 3

8

Right, the check keyword is missing, add it:

server 450adfs01 10.0.0.101:443 ssl verify none check check-sni myadfs.example.com sni ssl_fc_sni inter 3s rise 2 fall 3
9

Hi Lukas

This is quite ungrateful - sorry :frowning:

Now the check works, but it breaks ADFS

Med venlig hilsen/With best regards
Flemming Hougaard

SuperviseIT

10

What does the webinterface look like regarding checks? What does the log say?

You did replace the example domain myadfs.example.com with your real, proper domain, right?

11

Hi Lukas

When using the ladder config the log says: haproxy[19020]: backend ADFSBackend has no server available!

We are accessing the loadbalancing address on a fqdn called adfs.mydomain.com, and the backend server is called adfs01.mydomain.com & adfs02.mydomain.com and I’m replacing the example domain with the 2 last

/Flemming

Lukas Tribus via HAProxy community haproxy@discoursemail.com 7. marts 2019 18:30 >>>


lukastribus

    March 7

What does the webinterface look like regarding checks? What does the log say?

You did replace the example domain myadfs.example.com with your real, proper domain, right?

12

Are you sure all those health check configurations you made are really accurate?

/adfs/ls/IdpInitiatedSignon.aspx returning the string Sign\ in in the first octet? You are assuming a lot of things, you should break it down to small steps instead of starting with a huge and very complex configuration.

13

Hi Lukas

I tried to simplify as much I could

global
log /dev/log local0
log /dev/log local1 notice
maxconn 6000
tune.ssl.default-dh-param 2048
daemon
chroot /var/lib/haproxy
uid 99
gid 99
ssl-default-server-options force-tlsv12 no-tls-tickets
ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options force-tlsv12 no-tls-tickets
ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

defaults
log global
option tcplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

frontend ADFSFrontend
bind 10.0.0.100:443 ssl crt /etc/ssl/mycert.pem
mode tcp
default_backend ADFSBackend

backend ADFSBackend
mode tcp
balance roundrobin
option httpchk GET /adfs/ls/IdpInitiatedSignon.aspx HTTP/1.0\r\n
http-check expect string Sign\ in
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
server adfs01 10.0.0.101:443 ssl verify none check check-sni adfs01.mydomain.com sni ssl_fc_sni
server adfs02 10.0.0.102:443 ssl verify none check check-sni adfs02.mydomain.com sni ssl_fc_sni

And when I’m accessing the ADFS server on “https://adfs.mydomain.com/adfs/ls/IdpInitiatedSignon.aspx” I’ll get “ERR_EMPTY_RESPONSE”

And /var/log/haproxy.log shows the following:

2019-03-07T23:53:22.109767+01:00 adfsbal01 haproxy[20107]: Proxy ADFSFrontend started.
2019-03-07T23:53:22.110295+01:00 adfsbal01 haproxy[20107]: Proxy ADFSFrontend started.
2019-03-07T23:53:22.110604+01:00 adfsbal01 haproxy[20107]: Proxy ADFSBackend started.
2019-03-07T23:53:22.112213+01:00 adfsbal01 haproxy[20107]: Proxy ADFSBackend started.
2019-03-07T23:53:22.112531+01:00 adfsbal01 haproxy[20107]: Proxy Stats started.
2019-03-07T23:53:22.113180+01:00 adfsbal01 haproxy[20107]: Proxy Stats started.
2019-03-07T23:53:22.113452+01:00 adfsbal01 haproxy[20107]: Proxy stats started.
2019-03-07T23:53:22.113714+01:00 adfsbal01 haproxy[20107]: Proxy stats started.

And just to be sure, that I haven’t forgot anything - the haproxy is build on SLES 15, and has the following buildoptions:

haproxy -vv
HA-Proxy version 1.8.14-52e4d43 2018/09/20
Copyright 2000-2018 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = native
CC = gcc
CFLAGS = -O2 -march=native -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-format-truncation -Wno-null-dereference -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0i-fips 14 Aug 2018
Running on OpenSSL version : OpenSSL 1.1.0i-fips 14 Aug 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

And if you are in doubt - I really do appreciate your effort, thanks

/Flemming

14

If adfs.mydomain.com is pointing to the haproxy instances, but you don’t get any logs then probably you are not hitting the current haproxy instance. Double check that no obsolete haproxy instances are running in the background with old configurations.

15

Hi

Sorry the late replying from my side - it seemed I have had some misconfiguring in the logging part, and I have upgraded the server to latest SP level and the result is somewhat different now… when starting the haproxy comes with the following error:

Message from syslogd@localhost at Oct 2 20:09:43 …
haproxy[3959]: backend ADFSBackend has no server available!

and the config is:

global
log 127.0.0.1:514 local0 info
maxconn 6000
tune.ssl.default-dh-param 2048
daemon
chroot /var/lib/haproxy
uid 99
gid 99
ssl-default-server-options force-tlsv12 no-tls-tickets
ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options force-tlsv12 no-tls-tickets
ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

defaults
log global
option tcplog
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

frontend ADFSFrontend
bind 10.0.1.160:443 ssl crt /etc/haproxy/wildcard.cert.pem
mode http
http-request set-header X-MS-Forwarded-Client-IP %[src]
default_backend ADFSBackend

backend ADFSBackend
mode http
balance roundrobin
option httpchk GET /adfs/ls/IdpInitiatedSignon.aspx HTTP/1.0\r\n
option httpclose
option forwardfor header X-Client
http-check expect string Sign\ in
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
server adfssrv01 10.0.1.101:443 ssl verify none check check-sni adfs.domain.dk sni ssl_fc_sni inter 3s rise 2 fall 3
server adfssrv02 10.0.1.102:443 ssl verify none check check-sni adfs.domain.nyborg.dk sni ssl_fc_sni inter 3s rise 2 fall 3

and the error is:

2019-10-02T20:20:40+02:00 localhost haproxy[4102]: Proxy ADFSFrontend started.
2019-10-02T20:20:40+02:00 localhost haproxy[4102]: Proxy ADFSBackend started.
2019-10-02T20:20:40+02:00 localhost haproxy[4102]: Proxy Stats started.
2019-10-02T20:20:40+02:00 localhost haproxy[4102]: Proxy stats started.
2019-10-02T20:20:40+02:00 localhost haproxy[4103]: Server ADFSBackend/adfssrv01 is DOWN, reason: Layer7 invalid response, info: “HTTP content check did not match”, check duration: 9ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Message from syslogd@localhost at Oct 2 20:20:42 …
haproxy[4103]: backend ADFSBackend has no server available!
2019-10-02T20:20:42+02:00 localhost haproxy[4103]: Server ADFSBackend/adfssrv02 is DOWN, reason: Layer7 invalid response, info: “HTTP content check did not match”, check duration: 9ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2019-10-02T20:20:42+02:00 localhost haproxy[4103]: backend ADFSBackend has no server available!
2019-10-02T20:20:43+02:00 localhost haproxy[4103]: 10.0.255.3:54752 [02/Oct/2019:20:20:43.261] ADFSFrontend~ ADFSBackend/ 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 “GET /adfs/Proxy/webapplicationproxy/store?api-version=1 HTTP/1.1”
2019-10-02T20:20:43+02:00 localhost haproxy[4103]: 10.0.255.3:54753 [02/Oct/2019:20:20:43.288] ADFSFrontend~ ADFSBackend/ 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 “GET /adfs/Proxy/webapplicationproxy/store?api-version=1 HTTP/1.1”
2019-10-02T20:20:59+02:00 localhost haproxy[4103]: 10.0.255.4:54661 [02/Oct/2019:20:20:59.406] ADFSFrontend/1: SSL handshake failure
2019-10-02T20:21:08+02:00 localhost haproxy[4103]: 10.0.255.4:54662 [02/Oct/2019:20:21:08.025] ADFSFrontend~ ADFSBackend/ 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 “GET /adfs/Proxy/webapplicationproxy/store?api-version=1 HTTP/1.1”
2019-10-02T20:21:08+02:00 localhost haproxy[4103]: 10.0.255.4:54663 [02/Oct/2019:20:21:08.053] ADFSFrontend~ ADFSBackend/ 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 “GET /adfs/Proxy/webapplicationproxy/store?api-version=1 HTTP/1.1”
2019-10-02T20:21:13+02:00 localhost haproxy[4103]: 10.0.255.3:54754 [02/Oct/2019:20:21:13.259] ADFSFrontend~ ADFSBackend/ 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 “GET /adfs/Proxy/webapplicationproxy/store?api-version=1 HTTP/1.1”
2019-10-02T20:21:13+02:00 localhost haproxy[4103]: 10.0.255.3:54755 [02/Oct/2019:20:21:13.287] ADFSFrontend~ ADFSBackend/ 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 “GET /adfs/Proxy/webapplicationproxy/store?api-version=1 HTTP/1.1”

and I guess that’s have something to do with the SSL Handshake failure, but how?

/Flemming

16

Hi

I found an solution

In the ladder config the httpclose (and http-server-close) did break the communication with the ADFS server.

Furthermore I tried to change the httpchk to a GET on /adfs/probe… and If I do in eg. firefox I will get an 200… but Haproxy got an 503… but When returning to /adfs/ls/IdpInitiatedSignon.aspx, and changing the following “http-check expect status 200” - all started to function as expected.

Thanks for you time
/Flemming