No SSL on TCP Check

Help!
1

Hi HAProxy Experts!

Some Background:
we are using HAProxy in our Microservices environment running on Kubernetes. Some of the generated HAProxy config files have multiple backends and each of them hundreds of backend server.
Communication between our services is encrypted using TLS and we use HAProxy for SSL termination.

A server definition in the generated HAProxy config files look something like this:
server service_a:443 <ip-address>:443 id 1 check inter 30s rise 3 fall 2 ssl crt <crt-file> ca-file <ca-file> verify required verifyhost <service-fqdn>

With this setting, HAProxy does a complete SSL handshake to every server immediately after each restart or reload. This puts a huge CPU load on the sidecar container running HAProxy during the initial check (config property check).
I believe the fact that it is establishing a TLS connection for the check can be seen on the status page of HAProxy in the column LastChk with something line L6OK in 9ms while for non TLS connections doing a pure TCP check, it shows L4OK in 2ms.

Question:
We want to disable checks over SSL in our environment now and use pure TCP (layer 4) checks also for HTTPS backend server. We tried doing this by adding the option no-check-ssl to each server line, like the following from the above example:
server service_a:443 <ip-address>:443 id 1 check inter 30s rise 3 fall 2 ssl no-check-ssl crt <crt-file> ca-file <ca-file> verify required verifyhost <service-fqdn>.

However, adding this option does not seem to stop using SSL for checks since we still see L6OK in the stats page and not L4OK.

So, how can we disable using SSL for checks against backends using HTTPS?
Or did the option actually disable it but the display on the stats page is simply misleading?

Thank you in advance for looking into this question.

P.S. we are using HAProxy 1.8.17 and are in the process rolling out 1.9.10. We have only tested this with 1.8.17 yet.

2

Are you sure you don’t have option httpchk anywhere in the config? You need to remove health check configurations that will make haproxy do checks on higher layers.

Afaik haproxy does not ssl check by default. You only have to remove options, rather than add new ones.

no-check-ssl only reverts previous or default check-ssl settings. It does not guarantee that no SSL handshake will occur, if the handshake is required for example because it implies checking a HTTP response.

Share the default and backend section of your configuration please.

3

No httpchk anywhere in the config file and also no other option that would explicitly force checks on a higher level. All we have are mode http in each backend section.

Also, this sentence in the ssl option says, that when a server is on ssl it will also do the check over ssl:

But it is not clear how to switch off that health checks are sent in SSL? :cry:
We still want to do basic TCP checks though.

4

Sorry, didn’t see the request for the config before, so here it is:

global
        daemon
        user container
        group container
        maxconn 20000
        log     127.0.0.1 local1
        stats   socket /var/lib/haproxy/stats mode 666 level admin
        pidfile /var/run/haproxy.pid
        tune.ssl.cachesize 100000
        tune.vars.global-max-size 4194304
defaults
        log      global
        option   dontlognull
        option   contstats
        option   tcplog
        option   abortonclose
        option accept-invalid-http-request
        maxconn  20000
        retries  3
        timeout  connect 5s
        timeout  client  120m
        timeout  server  120m
        balance  roundrobin
        http-reuse aggressive
        option redispatch 10
        option prefer-last-server
        timeout http-keep-alive 1m
        option nolinger

listen stats
        bind :8081
        mode http
        stats enable
        stats uri /
        stats refresh 5s
        http-request set-log-level silent

frontend shared-frontend
        bind 127.0.0.1:80
        mode http
        option httplog
        acl is_service hdr_dom(host) -m reg ^service\.localhost(:80)?$
        use_backend my-backend if is_service
        option http-keep-alive

backend my-backend
        http-reuse aggressive
        acl original_host_header_exists req.hdr(X-Some-Custom-Header) -m found
        http-request replace-header Host (.*) %[req.hdr(X-Some-Custom-Header)] if original_host_header_exists
        mode http
        timeout server 120m
        timeout connect 5s
        server my-service-1:443 <ip-address>:443 id 1 check inter 30s rise 3 fall 2 no-check-ssl ssl crt <crt-file> ca-file <ca-file> verify required verifyhost my-service
        server my-service-2:443 <ip-address>:443 id 2 check inter 30s rise 3 fall 2 no-check-ssl ssl crt <crt-file> ca-file <ca-file> verify required verifyhost my-service
5

Indeed you’re right, it appears disabling SSL on health check wasn’t really thought of.

Theoretically you should be able to work around this limitation by re-specifying the health check port (check port 443):

server service_a:443 <ip-address>:443 id 1 check port 443 inter 30s rise 3 fall 2 ssl no-check-ssl crt <crt-file> ca-file <ca-file> verify required verifyhost <service-fqdn>

Please give it a try.

6

Hi @lukastribus!

That was indeed the key to the solution!

Thank you so much for sticking with me and helping to solve this puzzle.

Would be a good item for a FAQ, I guess.

Cheers,
-Uli

1 Like