SNI 503 error when using port 443 on backend

Help!
1

I am using SSL termination and SNI to two backend IIS servers. Haproxy version 1.6.9. The setup works for port 80 to the frontend and then port 80 to the backend. It used to work for port 443 to the fromtend and port 443 to the backend but now it throws 503 errors. If I do port 443 to the fromtend and port 80 to the backend it works but I need the backen traffic encrypted too. If I trace the packets I see a connection to the backend servers but then nothing happens. The sites have been verified to work on the backend ip over port 443 from a browser on a different system. I do not see why 443 stopped working for the proxy to backend connection.

Config for SNI is basically:

frontend shared-ip-frontend
bind 192.168.254.111:80
maxconn 40000

acl host_sample.com hdr_end(host) -i sample.com
use_backend sample.com if host_sample.com

#   Default backend for port 80

default_backend shared-ip-default-backend

frontend shared-ip-frontend-SSL
bind 192.168.254.111:443 ssl crt /etc/haproxy/certs-shared-ip
maxconn 40000
reqadd X-Forwarded-Proto:\ https

acl host_sample.com hdr_end(host) -i sample.com
use_backend sample.com-SSL if host_sample.com

# no default SSL backend

backend sample.com
option httpchk OPTIONS / HTTP/1.0\r\nHost:\ sample.com
stick-table type ip size 10k expire 20m
stick on src
server 192.168.7.111 192.168.7.111:80 check
server 192.168.6.111 192.168.6.111:80 check

backend sample.com-SSL
stick on src table sample.com
redirect scheme https if !{ ssl_fc }
server 192.168.7.111 192.168.7.111 ssl check port 80
server 192.168.6.111 192.168.6.111 ssl check port 80

2

Always specify the port in the backend servers, unless you need dynamic destination ports on your backend server. You already doing it for plaintext backend servers, you should do it for SSL enabled backends as well. You should also remove the “port 80” check configuration (you probably put it there because haproxy refused the configuration otherwise, but please specify the proper port instead). If this doesn’t help, remove the check configuration altogether, to force the server up.

Maybe you are running into some sort of validation issue, maybe the certificate of the backends expired? Do you use a CA to verify this or are using verify none somewhere? Please provide global and default sections as well.

Your log will tell whether a backend server was actually selected and used or not. So please the log line of that request.

So try something like this (and in case remove the check keyword):

server 192.168.7.111 192.168.7.111:443 ssl verify none check
server 192.168.6.111 192.168.6.111:443 ssl verify none check
3

Thanks for the information. I have changed the server line to have “…:443 ssl verify none” and the problem persists. It logs:

2017-12-04T13:48:06-06:00 localhost haproxy[8042]: xxx.xxx.xxx.xxx:1268 [04/Dec/2017:13:48:03.655] shared-ip-frontend-SSL~ sample.com-111-SSL/TESTSVR 53/3018/-1/-1/3071 503 212 - - SC-- 207/0/0/0/3 0/0 "GET / HTTP/1.1"
2017-12-04T13:48:09-06:00 localhost haproxy[8042]: xxx.xxx.xxx.xxx:1783 [04/Dec/2017:13:48:06.753] shared-ip-frontend-SSL~ sample.com-111-SSL/TESTSVR 3/3003/-1/-1/3006 503 212 - - SC-- 203/0/0/0/3 0/0 “GET /favicon.ico HTTP/1.1”

Here is my global section:

global
daemon
log 127.0.0.1 local2 info alert
nbproc 1
pidfile /var/run/haproxy.pid
maxconn 80000
tune.maxaccept -1
tune.maxrewrite 4096
tune.ssl.default-dh-param 4096
spread-checks 2

ssl-server-verify none
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-server-options no-sslv3
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

stats socket /var/lib/haproxy/stats.sock mode 600 level admin
stats timeout 2m
defaults
maxconn 2000
fullconn 2000
mode http
balance roundrobin
log global
option httplog
option forwardfor
option dontlognull
option dontlog-normal
option log-separate-errors
option http-server-close
option abortonclose
option redispatch
retries 3
default-server maxconn 2000 inter 15s fastinter 5s fall 2
timeout connect 3m
timeout client 4m
timeout server 4m
timeout queue 60s
timeout http-request 3m
timeout check 15s

4

The code is SC, which means the server refused the connection on port 443. Check if your backend servers have an SSL enabled socket on port 443 listening and that haproxy is allowed to access it.

On the haproxy box, try accessing them with curl:

curl -vvk https://192.168.7.111/
curl -vvk https://192.168.6.111/
5

The backend servers are listening on 443. I can pull up the sites from another computer’s browser if I use their ip directly.

From the haproxy server I get (similar for both):

[root@nsproxy haproxy]# curl -vvk https://192.168.6.111/

  • About to connect() to 192.168.6.111 port 443 (#0)
  • Trying 192.168.6.111… connected
  • Connected to 192.168.6.111 (192.168.6.111) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • NSS error -5961
  • Closing connection #0
  • SSL connect error
    curl: (35) SSL connect error

But for another non-SNI unique-ip site with SSL I get this from the haproxy:

[root@nsproxy haproxy]# curl -vvk https://192.168.4.144/

  • About to connect() to 192.168.4.144 port 443 (#0)
  • Trying 192.168.4.144… connected
  • Connected to 192.168.4.144 (192.168.4.144) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • Server certificate:
    subject: CN=www.sample2.com
    start date: Aug 23 00:00:00 2017 GMT
    expire date: Sep 22 23:59:59 2018 GMT
    common name: www.sample2.com
    issuer: CN=thawte DV SSL CA - G2,OU=Domain Validated SSL,O=“thawte, Inc.”,C=US

GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: 192.168.4.144
Accept: /

< HTTP/1.1 404 Not Found
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Mon, 04 Dec 2017 20:57:21 GMT
< Content-Length: 315
<

Not Found

Not Found

HTTP Error 404. The requested resource is not found.

* Connection #0 to host 192.168.4.144 left intact * Closing connection #0
6

So neither haproxy nor curl can access the HTTPS website. Clearly something is wrong with your backend.

Why do you say non-SNI? Are you implying that the backend server needs a particular SNI in the client hello?

7

We have diifferent sites configured on haproxy. Some have their own dedicated ip address so that are not using SNI and they work correctly over port 80 and 443 to the frontend and backend. We also have a single ip servicing multiple sites (via SNI) for port 80 only which work. But the one we have an issue with is a single ip address using SNI for multiple sites that we want both port 80 (which works) and port 443 (which throws 503 errors) to work. I hope that makes sense.

8

What is 192.168.6.111 and 192.168.7.111 exactly? Another haproxy instance? An Apache server? Does it require SNI on port 443 and reject the SSL connection otherwise (“strict-sni” in haproxy configuration)?

How does the curl command look like from the computer where it does open (in the browser)?

9

The two backend servers that server all the sites are Windows 2012 IIS servers. I do not have strict-sni defined. I have seen the initial connection over port 443 on the backend with netstat. I do not see anything logged in IIS for the attempted connection.

I will have to reconnect my laptop to check out the curl output.

Here is tcpdump info, if it helps, the haproxy server 192.168.4.254:

13:20:54.059938 IP 192.168.4.254.61956 > 192.168.7.111.https: Flags [SEW], seq 1456864300, win 14600, options [mss 1460,sackOK,TS val 304579458 ecr 0,nop,wscale 9], length 0
13:20:54.060239 IP 192.168.7.111.https > 192.168.4.254.61956: Flags [S.E], seq 1325677467, ack 1456864301, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 22326452 ecr 304579458], length 0
13:20:54.060252 IP 192.168.4.254.61956 > 192.168.7.111.https: Flags [.], ack 1, win 29, options [nop,nop,TS val 304579458 ecr 22326452], length 0
13:20:54.060281 IP 192.168.4.254.61956 > 192.168.7.111.https: Flags [P.], seq 1:190, ack 1, win 29, options [nop,nop,TS val 304579458 ecr 22326452], length 189
13:20:54.060568 IP 192.168.7.111.https > 192.168.4.254.61956: Flags [R.], seq 1, ack 190, win 0, length 0

10

Ok, also collect the output of the following openssl commands from the haproxy box please:

openssl s_client -connect 192.168.6.111:443

11

I get connected but I see the wrong cert. It gives the cert for the other SNI site.

[root@nsproxy bin]# openssl s_client -servername www.sample.com -connect 192.168.254.111:443
CONNECTED(00000003)
depth=2 C = US, O = “thawte, Inc.”, OU = Certification Services Division, OU = “© 2006 thawte, Inc. - For authorized use only”, CN = thawte Primary Root CA
verify return:1
depth=1 C = US, O = “thawte, Inc.”, OU = Domain Validated SSL, CN = thawte DV SSL CA - G2
verify return:1
depth=0 CN = www.sample2.com
verify return:1

Certificate chain
0 s:/CN=www.sample2.com
i:/C=US/O=thawte, Inc./OU=Domain Validated SSL/CN=thawte DV SSL CA - G2
1 s:/C=US/O=thawte, Inc./OU=Domain Validated SSL/CN=thawte DV SSL CA - G2
i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=© 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

Server certificate
-----BEGIN CERTIFICATE-----
MIIFjTCCBHWgAwIBAgIQN0v38bAXk5AkmE8+gi6eQDANBgkqhkiG9w0BAQsFADBj
…
KInJRrxi3YhOiTmttsitIMsxM+MExaiwoMbsNoyDAoec
-----END CERTIFICATE-----
subject=/CN=www.sample2.com
issuer=/C=US/O=thawte, Inc./OU=Domain Validated SSL/CN=thawte DV SSL CA - G2

No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits

SSL handshake has read 3345 bytes and written 397 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 6F944A5B94967F24B5A875A3DB88D8EFDC3312FB366108133112EDDCE96318C0
Session-ID-ctx:
Master-Key: A73688BF4DA9E8F2D1B011877E8AAB4E90B507F127820CAFFC804353878FDA1994B378439011DE3725287F2F4786362A
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - c4 ab 9b ba 16 8b 6d d8-d8 73 d4 d7 af 0f 95 6b …m…s…k
0010 - f2 a9 81 30 43 9b fa 33-58 a8 aa 37 7f de 97 16 …0C…3X…7…
0020 - d0 03 18 01 c4 0a 68 cf-eb 9f 32 2c 1e dd a3 9e …h…2,…
0030 - 4d 1f 22 66 67 cf 57 ed-c0 9b 92 9c 37 e1 ab 8a M.“fg.W…7…
0040 - 5d 10 96 5e 14 be c1 92-1f e9 79 4b bd 9a 04 ca ]…^…yK…
0050 - f2 a5 64 0e 82 f2 ed f6-bd 56 9c ef 61 21 76 c5 …d…V…a!v.
0060 - ba 5f e4 9c 64 f3 cc 53-d3 0f 81 81 22 99 8e 80 ._…d…S…”…
0070 - 5e 6d 9c 9e 2d 9f bd 60-55 74 5c 64 68 b2 93 75 ^m…-…Ut\dh..u 0080 - a2 d0 1e 42 c5 97 8e 84-5c ad 60 f1 6d 8f 3f 75 ...B....\..m.?u
0090 - c3 e1 20 80 06 21 f4 7a-c7 fd 92 a5 0e cc a6 12 … …!.z…
00a0 - 66 3c 7c 9d 58 c0 16 d2-d7 cf 4f 13 58 d4 d5 a0 f<|.X…O.X…

Start Time: 1512424586
Timeout   : 300 (sec)
Verify return code: 0 (ok)

xxx
HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html

400 Bad request

Your browser sent an invalid request. closed
12

So the IIS server DOES require SNI. This also means that https://192.168.6.11/ does not work in the browser (but only if you use the a correct hostname).

We need to set the SNI on backend SSL connection then:
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.2-sni

We can use the sni value from the frontend and pass it on:

server 192.168.7.111 192.168.7.111:443 ssl sni ssl_fc_sni check
server 192.168.6.111 192.168.6.111:443 ssl sni ssl_fc_sni check

Or use the host header directly:

server 192.168.7.111 192.168.7.111:443 ssl sni req.hdr(host) check
server 192.168.6.111 192.168.6.111:443 ssl sni req.hdr(host) check
1 Like
13

Lukas,
Thanks for that update on the server SNI settings. That cleared up the issue. I’m not sure what changed on the backend that made that necessary (I am not in control of the IIS servers) but it works as it did previously.\