Hi all,
I have a strange issue with haproxy after performed some package upgrades on my RHEL machine. Everything was fine but now when I try to get a resource (the second backend) i got 503 Service Unavailale.
I found error messages:
- haproxy[9190]: backend jboss-fe-bus has no server available!
- Server jboss-fe-bus/nodo1 is DOWN, reason: Layer6 invalid response, info: “SSL handshake failure”, check duration: 27ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Here’s my haproxy.cfg:
frontend myapp *:80
bind *:80
bind *:443 ssl crt /etc/pki/tls/private/mycertificate.pem
redirect scheme https if !{ ssl_fc }
option forwardfor
acl host_myapp1_bus hdr(host) -i app.mydomain.it
acl is_root path -i /
acl is_myapp1_bus path_beg /MYAPP
acl is_myapp3 path_beg /main-web
redirect code 302 location /MYAPP/ if is_root host_myapp1_bus
use_backend iis if host_myapp1_bus is_myapp1_bus
use_backend jboss-fe-bus if host_myapp1_bus is_myapp3
backend iis
balance roundrobin
cookie SERVERID insert indirect nocache
server nodo1 server01.domain.lan:443 weight 1 maxconn 100 check ssl verify none check cookie s1
backend jboss-fe-bus
balance roundrobin
server nodo1 server02.domain.lan:9443 weight 1 maxconn 100 check ssl verify none
The server “server02.domain.lan” is the same as haproxy.
Maybe new packages change something?
Thanks a lot,
Fabio
I assume we are talking about the openssl package that has been updated.
Can you tell what exact openssl package release you are running now, and what worked previously?
I assume the connection to the server01.domain.lan servers works fine?
Also please share the ouput of haproxy -vv.
Does the connection to server02 work when you remove the check keyword from the server line (so it is a health check problem)?
Hi,
thanks for your answer.
Please follow details:
haproxy -vv output:
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau willy@haproxy.org
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
###############################
openssl installed (from yum history) version:
Updated openssl-1:1.0.1e-60.el7_3.1.x86_64 @rhel-7-x86_64-latest
Update 1:1.0.2k-12.el7.x86_64 @rhel-7-server-rpms
Updated openssl-libs-1:1.0.1e-60.el7_3.1.x86_64 @rhel-7-x86_64-
################################
So the connection to server01 is fine and if i try to connect directly to https://server02:9443 it also works. I tried to remove check keyword and I got 502 Bad Gateway error.
Thanks,
Fabio
What does:
openssl version
openssl s_client -connect server02:9443
from your haproxy box report (exactly)?
HI,
my colleague is working with me looking for a fix. He downgraded openssl package to previous release with no effects.
Anyway here’s the output of commands:
openssl version
OpenSSL 1.0.2k-fips 26 Jan 2017
openssl s_client -connect server02:9443
CONNECTED(00000003)
depth=1 DC = XX, DC = XXX, CN = XXXX-CA
verify return:1
depth=0 C = X, ST = XX, L = XX, O = X X, OU = X, CN = server02
verify return:1
140477279786896:error:14082174:SSL routines:ssl3_check_cert_and_algorithm:dh key too small:s3_clnt.c:3636:
Certificate chain
0 s:/C=XX/ST=X/L=X/O=XX /OU=ICT/CN=server02.
i:/DC=XX/DC=XX/CN=XX-XX-CA
1 s:/DC=XX/DC=XX/CN=XX-XX-CA
i:/DC=XX/DC=XX/CN=XX-XX-CA
Server certificate
-----BEGIN CERTIFICATE-----
– OMISSIS –
-----END CERTIFICATE-----
subject=/C=X/ST=XX/L=XX/O=XX/OU=X/CN=server02
issuer=/DC=XX/DC=XX/CN=XX-XX-CA
No client certificate CA names sent
Server Temp Key: DH, 768 bits
SSL handshake has read 2996 bytes and written 7 bytes
New, (NONE), Cipher is (NONE)
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : 0000
Session-ID: 5B042E19AF89DE3105EA358130EC78CF188DF3CC15147275DA8110F3FE90E938
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1527000600
Timeout : 300 (sec)
Verify return code: 0 (ok)
I replace sensitive information with “XX”.
Thanks,
Fabio
Check this out:
140477279786896:error:14082174:SSL routines:ssl3_check_cert_and_algorithm:dh key too small:s3_clnt.c:3636:
and:
Server Temp Key: DH, 768 bits
You configured a tiny DH group in SSL backend server, that is causing the issue here as openssl refuses to connect to such a insecure configuration. Configure DH correctly or remove DH from the backend.
If you need a fast workaround from haproxy, try adding the following to your server line (server nodo1 server02.domain.lan:9443 ...), which disables DHE ciphers so you’d avoid hitting DH groups in the first place:
ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
Thanks @lukastribus, I tried your workaround but it doesn’t work.
How can I modify DH group in SSL backend in order to fix it?
Fabio
What’s your backend software? Does your certificate contain specific DH parameters?
Why do you need HTTPS anyway between haproxy and this backend server? You said it is on the same server as haproxy?
@lukastribus backend software is a JBoss server and there would need too much modify to switch to http.