Http backend checks failing with http/400; but curl to same url gives http/200 as expected

Help!
1

I am trying to enable http checks to my backend servers and I am a bit stuck.

My haproxy.cfg is:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE$
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend vIDM
        bind 10.10.1.83:80
        bind 10.10.1.83:443 ssl crt /etc/ssl/vidmlb.momusconsulting.com-chain-with-pk.pem
        mode http
        option http-server-close
        option forwardfor
        http-request add-header X-Forwarded-Proto https
        http-request add-header X-Forwarded-Port 443
        redirect scheme https if !{ ssl_fc }
        default_backend vIDM-be

backend vIDM-be
        mode http
        balance source
        cookie JSESSIONID rewrite nocache
        timeout check 2000
#       option httpchk
#       http-check connect port 443 ssl
#       http-check send meth GET uri /SAAS/API/1.0/REST/system/health/heartbeat ver HTTP/1.1
#       http-check expect status 200
        server mc-vidm-v-201a 10.10.1.85:443 check ssl verify none
        server mc-vidm-v-202a 10.10.1.86:443 check ssl verify none
        server mc-vidm-v-203a 10.10.1.87:443 check ssl verify none

listen Stats
        bind 10.10.1.80:8443 ssl crt /etc/ssl/mc-hapr-v-201a.momusconsulting.com-chain-with-pk.pem
        stats enable
        stats hide-version
        stats refresh 1s
        stats show-node
        stats auth stats:password
        stats uri /stats

When I un-comment the 4 hashed out lines in the ‘backend vIDM-be’ section and restart the haproxy service, I get Server vIDM-be/xx is DOWN, reason: Layer7 wrong status, code: 400.

Dec 01 13:58:54 mc-hapr-v-201a systemd[1]: Starting HAProxy Load Balancer…
Dec 01 13:58:54 mc-hapr-v-201a haproxy[3420]: [NOTICE] (3420) : New worker #1 (3434) forked
Dec 01 13:58:54 mc-hapr-v-201a systemd[1]: Started HAProxy Load Balancer.
Dec 01 13:58:54 mc-hapr-v-201a haproxy[3420]: [WARNING] (3434) : Server vIDM-be/mc-vidm-v-201a is DOWN, reason: Layer7 wrong status, code: 400, check duration: 18ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 01 13:58:55 mc-hapr-v-201a haproxy[3420]: [WARNING] (3434) : Server vIDM-be/mc-vidm-v-202a is DOWN, reason: Layer7 wrong status, code: 400, check duration: 19ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 01 13:58:55 mc-hapr-v-201a haproxy[3420]: [WARNING] (3434) : Server vIDM-be/mc-vidm-v-203a is DOWN, reason: Layer7 wrong status, code: 400, check duration: 17ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Dec 01 13:58:55 mc-hapr-v-201a haproxy[3420]: [NOTICE] (3434) : haproxy version is 2.4.9-1ppa1~bionic
Dec 01 13:58:55 mc-hapr-v-201a haproxy[3420]: [NOTICE] (3434) : path to executable is /usr/sbin/haproxy
Dec 01 13:58:55 mc-hapr-v-201a haproxy[3420]: [ALERT] (3434) : backend ‘vIDM-be’ has no server available!

When I curl the same url, I get a HTTP status 200/ok as expected.

root@mc-hapr-v-201a:/etc/haproxy# curl -vv https://mc-vidm-v-201a.momusconsulting.com/SAAS/API/1.0/REST/system/health/heartbeat
*   Trying 10.10.1.85...
* TCP_NODELAY set
* Connected to mc-vidm-v-201a.momusconsulting.com (10.10.1.85) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=GB; ST=Hampshire; L=Basingstoke; O=Momus Consulting; OU=Momus Labs; CN=vidmlb.momusconsulting.com
*  start date: Nov 30 15:04:59 2021 GMT
*  expire date: Nov 29 15:04:59 2026 GMT
*  subjectAltName: host "mc-vidm-v-201a.momusconsulting.com" matched cert's "mc-vidm-v-201a.momusconsulting.com"
*  issuer: DC=com; DC=momusconsulting; CN=MomusInterCA
*  SSL certificate verify ok.
> GET /SAAS/API/1.0/REST/system/health/heartbeat HTTP/1.1
> Host: mc-vidm-v-201a.momusconsulting.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200
< Cache-Control: no-cache, no-store, must-revalidate
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Strict-Transport-Security: max-age=31536000
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Content-Security-Policy: object-src 'none'; default-src blob: https: 'self' ; script-src 'unsafe-inline' 'unsafe-eval' https: 'self' ; style-src 'unsafe-inline' https: 'self'; img-src https: data: 'self'
< Set-Cookie: JSESSIONID=5D489A593C66497297D185655A4135A0; Path=/; Secure; HttpOnly
< P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
< Content-Type: text/plain;charset=ISO-8859-1
< Content-Length: 2
< Date: Wed, 01 Dec 2021 14:03:10 GMT
<
* Connection #0 to host mc-vidm-v-201a.momusconsulting.com left intact
okroot@mc-hapr-v-201a:/etc/haproxy#

I have been chasing my tail for an hour and I cannot see what I am doing wrong?

Any help will be greatly appreciated.

Cheers,
M

2

I assume that without Host header and SNI, your backend responds differently.

What does curl return without hostname:

curl -vvk https://10.10.1.85/SAAS/API/1.0/REST/system/health/heartbeat
3

Hello @lukastribus

Thank you for your help. I am not sure what a Host Header or SNI is/are.

Here are diff’s of both a curl -vv and curl -vvk for the backend server 201a via fqdn and IP.

curl -vv = haproxy - curl -vv - Diff Checker

curl -vvk = haproxy - curl -vvk - Diff Checker

The certificate /etc/ssl/vidmlb.momusconsulting.com-chain-with-pk.pem has the following SANs:

DNS.1 = mc-vidm-v-201a.momusconsulting.com
DNS.2 = mc-vidm-v-201a
DNS.3 = mc-vidm-v-202a.momusconsulting.com
DNS.4 = mc-vidm-v-202a
DNS.5 = mc-vidm-v-203a.momusconsulting.com
DNS.6 = mc-vidm-v-203a
DNS.7 = vidmlb.momusconsulting.com
DNS.8 = vidmlb
IP.1 = 10.10.1.83
IP.2 = 10.10.1.84
IP.3 = 10.10.1.85
IP.4 = 10.10.1.86
IP.5 = 10.10.1.87

Please let me know if you would like to see any other output.

Cheers,
M

4

I’m afraid we need to dig deeper. I can see the backend is responding without a reason phrase (HTTP/1.1 200 instead of HTTP/1.1 200 OK), but the reason phrase is optional as per the RFC (though the space after the status code is not); so this is not enough to be able to conclude what happens here.

Is the health checking endpoint also available without SSL, on a plaintext port? In this case, move the check to that and capture the health check traffic fully with tcpdump (for example tcpdump -i eth0 -pns0 -w health-check.cap host 10.10.1.85 and tcp port 80).

If you only get this health check endpoint via HTTPS, then we need to capture the health check via HTTPS and decrypt it with the private key of the certificate (making sure we don’t use a Forward Secrecy cipher).

But I suggest you take a look at the backend server logs first: are they indicating a 400 response to those health check requests, or 200? Any other information you can get from the backend server logs would be useful.

Then try plaintext health-checking, if possible. This probably fails the same way, but we can capture the health check traffic without decrypting SSL, which is a complicated process.

If none of this is useful, add this to the global section to restrict SSL to non-FS ciphers on the backend:

ssl-default-server-ciphers AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA

Capture the health-check traffic via SSL and decrypt it in wireshark with the private key of the certificate. For example see: Wireshark Tutorial: Decrypting HTTPS Traffic (Includes SSL and TLS)

5

Hi @lukastribus,

When I added that ssl-default-server-ciphers setting to the global config and restarted haproxy service (with the health checks still disabled), the 3 backend servers were immediately put in the DOWN state.

I removed the ssl-default-server-ciphers setting and was able to capture the failing health check over http/80 for backend node 201a with the following config:

        option httpchk
        http-check connect port 80
        http-check send meth GET uri /SAAS/API/1.0/REST/system/health/heartbeat ver HTTP/1.1
        http-check expect status 200

The .pcap file is in my GitHub repo. I hope it is what you needed.

I have been unable to find any correlating logs on the backend system yet - I will keep digging.

Let me know if you need anything else.

Cheers,
M

6

Yeah, that helps a lot (you no longer need to play with SSL settings, that would just be required if we would need to decrypt SSL, but fortunately you can reproduce this on a plaintext port).

The 400 error percieved by haproxy from the backend server is real, it looks like the backend server doesn’t like the request from haproxy:

grafik

Lets try with curl. Can you run the following request and check if the return status is 400 or 200. If it is 400, try removing the last -H parameter until you see a 200 response, then we will know what the backend server doesn’t like about the request.

curl -vv https://10.10.1.85/SAAS/API/1.0/REST/system/health/heartbeat \
-H "User-Agent:" -H "Accept:" -H "content-length: 0" -H "connection: close" -H "Host:"
7

Morning @lukastribus

Thank you. Curl test results below.

TL;DR; It returns a 400 response when the -H "Host:" parameter is specified.

400 Response:

root@mc-hapr-v-201a:/etc/haproxy# curl -vv https://10.10.1.85/SAAS/API/1.0/REST/system/health/heartbeat -H "User-Agent:" -H "Accept:" -H "content-length: 0" -H "connection: close" -H "Host:"
*   Trying 10.10.1.85...
* TCP_NODELAY set
* Connected to 10.10.1.85 (10.10.1.85) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=GB; ST=Hampshire; L=Basingstoke; O=Momus Consulting; OU=Momus Labs; CN=vidmlb.momusconsulting.com
*  start date: Nov 30 15:04:59 2021 GMT
*  expire date: Nov 29 15:04:59 2026 GMT
*  subjectAltName: host "10.10.1.85" matched cert's IP address!
*  issuer: DC=com; DC=momusconsulting; CN=MomusInterCA
*  SSL certificate verify ok.
> GET /SAAS/API/1.0/REST/system/health/heartbeat HTTP/1.1
> content-length: 0
> connection: close
>
< HTTP/1.1 400
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 762
< Date: Thu, 02 Dec 2021 08:52:06 GMT
< Connection: close
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
<!doctype html><html lang="en"><head><title>HTTP Status 400 – Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 – Bad Request</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).</p><hr class="line" /><h3>Apache Tomcat/8.5.63</h3></body></html>root@mc-hapr-v-201a:/etc/haproxy#
root@mc-hapr-v-201a:/etc/haproxy#

200 Response:

root@mc-hapr-v-201a:/etc/haproxy# curl -vv https://10.10.1.85/SAAS/API/1.0/REST/system/health/heartbeat -H "User-Agent:" -H "Accept:" -H "content-length: 0" -H "connection: close"
*   Trying 10.10.1.85...
* TCP_NODELAY set
* Connected to 10.10.1.85 (10.10.1.85) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=GB; ST=Hampshire; L=Basingstoke; O=Momus Consulting; OU=Momus Labs; CN=vidmlb.momusconsulting.com
*  start date: Nov 30 15:04:59 2021 GMT
*  expire date: Nov 29 15:04:59 2026 GMT
*  subjectAltName: host "10.10.1.85" matched cert's IP address!
*  issuer: DC=com; DC=momusconsulting; CN=MomusInterCA
*  SSL certificate verify ok.
> GET /SAAS/API/1.0/REST/system/health/heartbeat HTTP/1.1
> Host: 10.10.1.85
> content-length: 0
> connection: close
>
< HTTP/1.1 200
< Cache-Control: no-cache, no-store, must-revalidate
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Strict-Transport-Security: max-age=31536000
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Content-Security-Policy: object-src 'none'; default-src blob: https: 'self' ; script-src 'unsafe-inline' 'unsafe-eval' https: 'self' ; style-src 'unsafe-inline' https: 'self'; img-src https: data: 'self'
< Set-Cookie: JSESSIONID=3448AAD7C57B54A7D15C72DC01E6DDEA; Path=/; Secure; HttpOnly
< P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
< Content-Type: text/plain;charset=ISO-8859-1
< Content-Length: 2
< Date: Thu, 02 Dec 2021 08:53:56 GMT
< Connection: close
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
okroot@mc-hapr-v-201a:/etc/haproxy#
root@mc-hapr-v-201a:/etc/haproxy#

Here is a diff of both outputs above - haproxy - curl diffs with and with out -H "Host:" parameter - Diff Checker

Can anything be done to overcome or work around this?

Please let me know if you need anymore info.

Thanks again.
M

8

Ok, then we need to add a host header to the health check

    http-check send meth GET uri /SAAS/API/1.0/REST/system/health/heartbeat ver HTTP/1.1 hdr host example.org
1 Like
9

:partying_face: :partying_face: Yay, that worked!!!

Thank you for all of your help @lukastribus I really appreciate it.

My running haproxy.cfg now looks like this:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE$
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend vIDM
        bind 10.10.1.83:80
        bind 10.10.1.83:443 ssl crt /etc/ssl/vidmlb.momusconsulting.com-chain-with-pk.pem
        mode http
        option http-server-close
        option forwardfor
        http-request add-header X-Forwarded-Proto https
        http-request add-header X-Forwarded-Port 443
        redirect scheme https if !{ ssl_fc }
        default_backend vIDM-be

backend vIDM-be
        mode http
        balance source
        cookie JSESSIONID rewrite nocache
        timeout check 2000
        option httpchk
        http-check connect port 443 ssl
        http-check send meth GET uri /SAAS/API/1.0/REST/system/health/heartbeat ver HTTP/1.1 hdr host vidmlb.momusconsulting.com
        http-check expect status 200
        server mc-vidm-v-201a 10.10.1.85:443 check ssl verify none
        server mc-vidm-v-202a 10.10.1.86:443 check ssl verify none
        server mc-vidm-v-203a 10.10.1.87:443 check ssl verify none

listen Stats
        bind 10.10.1.80:8443 ssl crt /etc/ssl/mc-hapr-v-201a.momusconsulting.com-chain-with-pk.pem
        stats enable
        stats hide-version
        stats refresh 1s
        stats show-node
        stats auth stats:password
        stats uri /stats
        stats show-modules
        stats show-legends

Stats Report looks great too.

HAProxy Stats with working L7 checks
HAProxy Stats with working L7 checks1667×628 57.5 KB

Thank you again,

Cheers,
M

1 Like