SNI 503 error when using port 443 on backend

I am using SSL termination and SNI to two backend IIS servers. Haproxy version 1.6.9. The setup works for port 80 to the frontend and then port 80 to the backend. It used to work for port 443 to the fromtend and port 443 to the backend but now it throws 503 errors. If I do port 443 to the fromtend and port 80 to the backend it works but I need the backen traffic encrypted too. If I trace the packets I see a connection to the backend servers but then nothing happens. The sites have been verified to work on the backend ip over port 443 from a browser on a different system. I do not see why 443 stopped working for the proxy to backend connection.

Config for SNI is basically:

frontend shared-ip-frontend
maxconn 40000

acl hdr_end(host) -i
use_backend if

#   Default backend for port 80

default_backend shared-ip-default-backend

frontend shared-ip-frontend-SSL
bind ssl crt /etc/haproxy/certs-shared-ip
maxconn 40000
reqadd X-Forwarded-Proto:\ https

acl hdr_end(host) -i
use_backend if

# no default SSL backend

option httpchk OPTIONS / HTTP/1.0\r\nHost:\
stick-table type ip size 10k expire 20m
stick on src
server check
server check

stick on src table
redirect scheme https if !{ ssl_fc }
server ssl check port 80
server ssl check port 80

Always specify the port in the backend servers, unless you need dynamic destination ports on your backend server. You already doing it for plaintext backend servers, you should do it for SSL enabled backends as well. You should also remove the “port 80” check configuration (you probably put it there because haproxy refused the configuration otherwise, but please specify the proper port instead). If this doesn’t help, remove the check configuration altogether, to force the server up.

Maybe you are running into some sort of validation issue, maybe the certificate of the backends expired? Do you use a CA to verify this or are using verify none somewhere? Please provide global and default sections as well.

Your log will tell whether a backend server was actually selected and used or not. So please the log line of that request.

So try something like this (and in case remove the check keyword):

server ssl verify none check
server ssl verify none check

Thanks for the information. I have changed the server line to have “…:443 ssl verify none” and the problem persists. It logs:

2017-12-04T13:48:06-06:00 localhost haproxy[8042]: [04/Dec/2017:13:48:03.655] shared-ip-frontend-SSL~ 53/3018/-1/-1/3071 503 212 - - SC-- 207/0/0/0/3 0/0 "GET / HTTP/1.1"
2017-12-04T13:48:09-06:00 localhost haproxy[8042]: [04/Dec/2017:13:48:06.753] shared-ip-frontend-SSL~ 3/3003/-1/-1/3006 503 212 - - SC-- 203/0/0/0/3 0/0 “GET /favicon.ico HTTP/1.1”

Here is my global section:

log local2 info alert
nbproc 1
pidfile /var/run/
maxconn 80000
tune.maxaccept -1
tune.maxrewrite 4096
tune.ssl.default-dh-param 4096
spread-checks 2

ssl-server-verify none
ssl-default-bind-options no-sslv3
ssl-default-server-options no-sslv3

stats socket /var/lib/haproxy/stats.sock mode 600 level admin
stats timeout 2m
maxconn 2000
fullconn 2000
mode http
balance roundrobin
log global
option httplog
option forwardfor
option dontlognull
option dontlog-normal
option log-separate-errors
option http-server-close
option abortonclose
option redispatch
retries 3
default-server maxconn 2000 inter 15s fastinter 5s fall 2
timeout connect 3m
timeout client 4m
timeout server 4m
timeout queue 60s
timeout http-request 3m
timeout check 15s

The code is SC, which means the server refused the connection on port 443. Check if your backend servers have an SSL enabled socket on port 443 listening and that haproxy is allowed to access it.

On the haproxy box, try accessing them with curl:

curl -vvk
curl -vvk

The backend servers are listening on 443. I can pull up the sites from another computer’s browser if I use their ip directly.

From the haproxy server I get (similar for both):

[root@nsproxy haproxy]# curl -vvk

  • About to connect() to port 443 (#0)
  • Trying… connected
  • Connected to ( port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • NSS error -5961
  • Closing connection #0
  • SSL connect error
    curl: (35) SSL connect error

But for another non-SNI unique-ip site with SSL I get this from the haproxy:

[root@nsproxy haproxy]# curl -vvk

  • About to connect() to port 443 (#0)
  • Trying… connected
  • Connected to ( port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • Server certificate:
    start date: Aug 23 00:00:00 2017 GMT
    expire date: Sep 22 23:59:59 2018 GMT
    common name:
    issuer: CN=thawte DV SSL CA - G2,OU=Domain Validated SSL,O=“thawte, Inc.”,C=US

GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Accept: /

< HTTP/1.1 404 Not Found
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Mon, 04 Dec 2017 20:57:21 GMT
< Content-Length: 315

Not Found

Not Found

HTTP Error 404. The requested resource is not found.

* Connection #0 to host left intact * Closing connection #0

So neither haproxy nor curl can access the HTTPS website. Clearly something is wrong with your backend.

Why do you say non-SNI? Are you implying that the backend server needs a particular SNI in the client hello?

We have diifferent sites configured on haproxy. Some have their own dedicated ip address so that are not using SNI and they work correctly over port 80 and 443 to the frontend and backend. We also have a single ip servicing multiple sites (via SNI) for port 80 only which work. But the one we have an issue with is a single ip address using SNI for multiple sites that we want both port 80 (which works) and port 443 (which throws 503 errors) to work. I hope that makes sense.

What is and exactly? Another haproxy instance? An Apache server? Does it require SNI on port 443 and reject the SSL connection otherwise (“strict-sni” in haproxy configuration)?

How does the curl command look like from the computer where it does open (in the browser)?

The two backend servers that server all the sites are Windows 2012 IIS servers. I do not have strict-sni defined. I have seen the initial connection over port 443 on the backend with netstat. I do not see anything logged in IIS for the attempted connection.

I will have to reconnect my laptop to check out the curl output.

Here is tcpdump info, if it helps, the haproxy server

13:20:54.059938 IP > Flags [SEW], seq 1456864300, win 14600, options [mss 1460,sackOK,TS val 304579458 ecr 0,nop,wscale 9], length 0
13:20:54.060239 IP > Flags [S.E], seq 1325677467, ack 1456864301, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 22326452 ecr 304579458], length 0
13:20:54.060252 IP > Flags [.], ack 1, win 29, options [nop,nop,TS val 304579458 ecr 22326452], length 0
13:20:54.060281 IP > Flags [P.], seq 1:190, ack 1, win 29, options [nop,nop,TS val 304579458 ecr 22326452], length 189
13:20:54.060568 IP > Flags [R.], seq 1, ack 190, win 0, length 0

Ok, also collect the output of the following openssl commands from the haproxy box please:

openssl s_client -connect

I get connected but I see the wrong cert. It gives the cert for the other SNI site.

[root@nsproxy bin]# openssl s_client -servername -connect
depth=2 C = US, O = “thawte, Inc.”, OU = Certification Services Division, OU = “© 2006 thawte, Inc. - For authorized use only”, CN = thawte Primary Root CA
verify return:1
depth=1 C = US, O = “thawte, Inc.”, OU = Domain Validated SSL, CN = thawte DV SSL CA - G2
verify return:1
depth=0 CN =
verify return:1

Certificate chain
0 s:/
i:/C=US/O=thawte, Inc./OU=Domain Validated SSL/CN=thawte DV SSL CA - G2
1 s:/C=US/O=thawte, Inc./OU=Domain Validated SSL/CN=thawte DV SSL CA - G2
i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=© 2006 thawte, Inc. - For authorized use only/CN=thawte Primary Root CA

Server certificate

issuer=/C=US/O=thawte, Inc./OU=Domain Validated SSL/CN=thawte DV SSL CA - G2

No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits

SSL handshake has read 3345 bytes and written 397 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 6F944A5B94967F24B5A875A3DB88D8EFDC3312FB366108133112EDDCE96318C0
Master-Key: A73688BF4DA9E8F2D1B011877E8AAB4E90B507F127820CAFFC804353878FDA1994B378439011DE3725287F2F4786362A
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - c4 ab 9b ba 16 8b 6d d8-d8 73 d4 d7 af 0f 95 6b …m…s…k
0010 - f2 a9 81 30 43 9b fa 33-58 a8 aa 37 7f de 97 16 …0C…3X…7…
0020 - d0 03 18 01 c4 0a 68 cf-eb 9f 32 2c 1e dd a3 9e …h…2,…
0030 - 4d 1f 22 66 67 cf 57 ed-c0 9b 92 9c 37 e1 ab 8a M.“fg.W…7…
0040 - 5d 10 96 5e 14 be c1 92-1f e9 79 4b bd 9a 04 ca ]…^…yK…
0050 - f2 a5 64 0e 82 f2 ed f6-bd 56 9c ef 61 21 76 c5 …d…V…a!v.
0060 - ba 5f e4 9c 64 f3 cc 53-d3 0f 81 81 22 99 8e 80 ._…d…S…”…
0070 - 5e 6d 9c 9e 2d 9f bd 60-55 74 5c 64 68 b2 93 75 ^m…-…Ut\dh..u 0080 - a2 d0 1e 42 c5 97 8e 84-5c ad 60 f1 6d 8f 3f 75 ...B....\..m.?u
0090 - c3 e1 20 80 06 21 f4 7a-c7 fd 92 a5 0e cc a6 12 … …!.z…
00a0 - 66 3c 7c 9d 58 c0 16 d2-d7 cf 4f 13 58 d4 d5 a0 f<|.X…O.X…

Start Time: 1512424586
Timeout   : 300 (sec)
Verify return code: 0 (ok)

HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html

400 Bad request

Your browser sent an invalid request. closed

So the IIS server DOES require SNI. This also means that does not work in the browser (but only if you use the a correct hostname).

We need to set the SNI on backend SSL connection then:

We can use the sni value from the frontend and pass it on:

server ssl sni ssl_fc_sni check
server ssl sni ssl_fc_sni check

Or use the host header directly:

server ssl sni req.hdr(host) check
server ssl sni req.hdr(host) check
1 Like

Thanks for that update on the server SNI settings. That cleared up the issue. I’m not sure what changed on the backend that made that necessary (I am not in control of the IIS servers) but it works as it did previously.\