Help with SSL config with 1 root cert but mutiple subdomains listening on port 443

Hi ,
I am new to Haproxy.
I have installed Ha-proxy version 2.1.4 stable version on Ubuntu.
I have a working config with TCP mode but my applications need https for Java api calls which fail with Handshake error using TCP mode i.e needs SSL certs .

I wanted to know if the below configuration is possible.Are there any other solution for my scenario?

frontend https_front

   bind *:443 ssl crt /etc/ssl/root.example.com.pem (it has both cert and private key)
   mode http
  ### the below acls are not correct.Just added for explaining my scenario###
  acl if  app1.example.com use backend be_app1
  acl if app2.example.com use backend be_app2
  acl if app3.example.com use backend be_app3

backend be_app1
balance source
mode http
server ser1 10.1.101.10:8443 ssl check verify none
server ser2 10.1.101.11:8443 ssl check verify none

backend be_app2
balance source
mode http
server ser1 10.1.101.10:8445 ssl check verify none
server ser2 10.1.101.11:8445 ssl check verify none

backend be_app3
balance source
mode http
server ser1 10.1.101.10:8447 ssl check verify none
server ser2 10.1.101.11:8447 ssl check verify none

Thanks,
Sri

If your question is: can haproxy reencrypt SSL on the backend, then yes, it can.

Not sure though I understood your requirement correctly though.

Hi Lukastribus,

My apologies for not being clear with my request.With frontend and backend using TCP i can use the req.ssl_sni acl to route the apps to appropriate servers in the backend.
If the port 443(frontend http(s) mode) is used for all apps(app1.example.com,app2.example.com,app3.example.com) using sub -domains and 1 root certficate (*.example.com) ,how will you route the frontend request to the appropriate backend i.e. is there any acl like chekcing the req.ssl_sni for http?

Thanks,
Sri

I have no idea what you are asking.

So you are routing based on SNI in TCP mode and that is working fine?

That’s not a root certificate, that’s a wildcard certificate. And it cannot overlap with the other certificates, or you will have issues with browser reusing the wrong sessions.

*.example.com overlaps with app1.example.com

What does this question mean? Can you first explain the actual problem that you are trying to solve in a few sentences?

Hi Lukastribus,

Thanks again for your response.
Again my apologies for not being clear.

The below TCP (SSL Passthrough) config works fine when the applications are accessed from browser example: https://app1.example.com .
The application also has REST api calls which are failing with error SSL handshake error with this TCP mode config.

I am assuming the application API calls are failing due to TCP mode ie. expecting SSL in Haproxy. How would I configure the same using https using the wildcard certificates (as mentioned before) or is there an option to fix this SSL handshake error with TCP config ?

########################################

HA-Proxy version 2.1.4-1ppa1~bionic 2020/04/12 - https://haproxy.org/

Status: stable branch - will stop receiving fixes around Q1 2021.

Known bugs: http://www.haproxy.org/bugs/bugs-2.1.4.html

Build options :

TARGET = linux-glibc

CPU = generic

CC = gcc

CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-DKPuPX/haproxy-2.1.4=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference

OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :

bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=1).

Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018

Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018

OpenSSL library supports TLS extensions : yes

OpenSSL library supports SNI : yes

OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3

Built with Lua version : Lua 5.3.3

Built with network namespace support.

Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Built with PCRE2 version : 10.31 2018-02-12

PCRE2 library supports JIT : yes

Encrypted password support via crypt(3): yes

Built with zlib version : 1.2.11

Running on zlib version : 1.2.11

Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)

Built with the Prometheus exporter as a service

Available polling systems :

  epoll : pref=300,  test result OK

   poll : pref=200,  test result OK

 select : pref=150,  test result OK

Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :

(protocols marked as cannot be specified using ‘proto’ keyword)

          h2 : mode=HTTP       side=FE|BE     mux=H2

        fcgi : mode=HTTP       side=BE        mux=FCGI

   <default> : mode=HTTP       side=FE|BE     mux=H1

   <default> : mode=TCP        side=FE|BE     mux=PASS

Available services :

    prometheus-exporter

Available filters :

    [SPOE] spoe

    [CACHE] cache

    [FCGI] fcgi-app

    [TRACE] trace

    [COMP] compression

########################################

defaults

log global

mode tcp

option tcplog

option dontlognull

option log-health-checks

timeout connect 5000

timeout client 50000

timeout server 50000

frontend stats

 bind *:8404

 mode http

stats enable

stats uri /stats

stats refresh 10s

stats hide-version

frontend http_front

bind *:443

#bind *:443 ssl crt /etc/ssl/ppllc/example.com.pem

option tcplog

acl network_allowed src -f /etc/haproxy/acls/whitelisted_cidr.acl

tcp-request connection reject if !network_allowed

#acl tls req.ssl_hello_type 1

tcp-request inspect-delay 5s

#tcp-request content accept if tls

tcp-request content accept if { req_ssl_hello_type 1 }

acl host_app1 req.ssl_sni -i app1.example.com

acl host_app2 req.ssl_sni -i app2.example.com

acl host_app3 req.ssl_sni -i app3.example.com

use_backend be_app1 if host_app1

use_backend be_app2 if host_app2

use_backend be_app3 if host_app3

backend be_app1

    balance source



    # maximum SSL session ID length is 32 bytes.

    stick-table type binary len 32 size 30k expire 30m

    #stick-table type ip size 1m expire 1h

    stick on src



    acl clienthello req_ssl_hello_type 1

    acl serverhello rep_ssl_hello_type 2



    # use tcp content accepts to detects ssl client and server hello.

    tcp-request inspect-delay 5s

    tcp-request content accept if clienthello



    # no timeout on response inspect delay by default.

    tcp-response content accept if serverhello



    # SSL session ID (SSLID) may be present on a client or server hello.

    # Its length is coded on 1 byte at offset 43 and its value starts

    # at offset 44.

    # Match and learn on request if client hello.

    stick on payload_lv(43,1) if clienthello



    # Learn on response if server hello.

    stick store-response payload_lv(43,1) if serverhello



    option tcp-check



    server serv1_8387 10.23.19.166:8387 check fall 3 rise 2

    server serv2_8387 10.23.19.167:8387 check fall 3 rise 2

backend be_app2

    balance source



    # maximum SSL session ID length is 32 bytes.

    stick-table type binary len 32 size 30k expire 30m

    #stick-table type ip size 1m expire 1h

    stick on src



    acl clienthello req_ssl_hello_type 1

    acl serverhello rep_ssl_hello_type 2



    # use tcp content accepts to detects ssl client and server hello.

    tcp-request inspect-delay 5s

    tcp-request content accept if clienthello



    # no timeout on response inspect delay by default.

    tcp-response content accept if serverhello



    # SSL session ID (SSLID) may be present on a client or server hello.

    # Its length is coded on 1 byte at offset 43 and its value starts

    # at offset 44.

    # Match and learn on request if client hello.

    stick on payload_lv(43,1) if clienthello



    # Learn on response if server hello.

    stick store-response payload_lv(43,1) if serverhello



    option tcp-check



    server serv3_8387 10.23.19.166:8400 check fall 3 rise 2

    server serv4_8387 10.23.19.167:8400 check fall 3 rise 2

backend be_app3

    balance source



    # maximum SSL session ID length is 32 bytes.

    stick-table type binary len 32 size 30k expire 30m

    #stick-table type ip size 1m expire 1h

    stick on src



    acl clienthello req_ssl_hello_type 1

    acl serverhello rep_ssl_hello_type 2



    # use tcp content accepts to detects ssl client and server hello.

    tcp-request inspect-delay 5s

    tcp-request content accept if clienthello



    # no timeout on response inspect delay by default.

    tcp-response content accept if serverhello



    # SSL session ID (SSLID) may be present on a client or server hello.

    # Its length is coded on 1 byte at offset 43 and its value starts

    # at offset 44.

    # Match and learn on request if client hello.

    stick on payload_lv(43,1) if clienthello



    # Learn on response if server hello.

    stick store-response payload_lv(43,1) if serverhello



    option tcp-check



    server serv1_8387 10.23.19.166:8500 check fall 3 rise 2

    server serv2_8387 10.23.19.167:8500 check fall 3 rise 2

#######################

Thanks ,
Sri

The actual error message i see in the API calls using TCP mode “Remote host closed connection during Handshake”

Thanks,
Sri

Find below the complete error stack from api call

] [com.ncsts.view.bean.ApiTestBean]:138 - Remote host closed connection during handshake

javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

           at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808)

           at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1112)

           at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623)

           at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)

           at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:131)

           at org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:138)

           at org.apache.http.impl.io.ContentLengthOutputStream.flush(ContentLengthOutputStream.java:102)

           at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:122)

           at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96)

           at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)

           at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:120)

           at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:264)

           at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:224)

           at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:255)

           at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)

           at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:647)

           at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464)

           at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

           at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:941)

           at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:919)

           at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:910)

           at com.ncsts.view.bean.ApiTestBean.post(ApiTestBean.java:132)

           at com.ncsts.view.bean.ApiTestBean.submitAction(ApiTestBean.java:107)

           at com.ncsts.view.bean.FilePreferenceBackingBean.verifyPinPointAPIActionListener(FilePreferenceBackingBean.java:4195)

           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

           at java.lang.reflect.Method.invoke(Method.java:597)

           at org.apache.el.parser.AstValue.invoke(AstValue.java:172)

           at org.apache.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)

           at com.sun.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:68)

           at javax.faces.event.MethodExpressionActionListener.processAction(MethodExpressionActionListener.java:99)

           at javax.faces.event.ActionEvent.processListener(ActionEvent.java:88)

           at javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:760)

           at javax.faces.component.UICommand.broadcast(UICommand.java:372)

           at org.ajax4jsf.component.AjaxViewRoot.processEvents(AjaxViewRoot.java:329)

           at org.ajax4jsf.component.AjaxViewRoot.broadcastEventsForPhase(AjaxViewRoot.java:304)

           at org.ajax4jsf.component.AjaxViewRoot.processPhase(AjaxViewRoot.java:261)

           at org.ajax4jsf.component.AjaxViewRoot.processApplication(AjaxViewRoot.java:474)

           at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:82)

           at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100)

           at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)

           at org.apache.myfaces.custom.ppr.PPRLifecycleWrapper.execute(PPRLifecycleWrapper.java:68)

           at javax.faces.webapp.FacesServlet.service(FacesServlet.java:265)

           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

           at org.apache.myfaces.webapp.filter.ExtensionsFilter.doFilter(ExtensionsFilter.java:147)

           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

           at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:206)

           at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)

           at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:388)

           at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:515)

           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

           at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

           at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

           at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)

           at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)

           at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

           at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)

           at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

           at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)

           at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849)

           at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)

           at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454)

           at java.lang.Thread.run(Thread.java:619)

Caused by: java.io.EOFException: SSL peer shut down incorrectly

           at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)

           at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)

           ... 66 more

Let’s take a look at those assumptions first.

You have a configuration that requires SNI. Are you positive that your REST API calls from the application actually send SNI? Did you capture the client_hellos and check?

I will check with the application developer if the API calls send SNI and update you .

Thanks
Sri

I suggest you capture the SSL handshake and check for yourself.

Can you let me know how to capture the SSL handshake?

Thanks,
Sri

on the haproxy instance, capture the traffic with tcpdump:

tcpdump -nps0 -i eth0 -w capturedSSL.cap tcp port 443

And then you can review it in wireshark.

You can also try ssldump, heres a article about that:

https://devcentral.f5.com/s/articles/troubleshooting-tls-problems-with-ssldump

Thank you very much.

Regards,
Sri

Lukastribus,

The API calls were not using SNI.I learn’t a lot from your feedback and suggestions.

Thanks again for all your help!

Regards,
Srini

1 Like

Lukastribus,

The API call program runs on older version of Java which cannot enable SNI.Is there any other solutions/config in Haproxy for using without SNI i.e.
wildcard certs configured to single IP and different applications running on sub-domains listening on port 443 frontend ?

Thanks,
Sri

Lukastribus,

Will the below config work for wildcard cert and multiple subdomains i.e using hrd_sub instead of SNI(in TCP mode)?

frontend https_front

bind *:8484 ssl crt /etc/ssl/example.com.pem

mode http

log global

option log-separate-errors

option logasap

option forwardfor

http-request set-header X-Forwarded-Proto https if { ssl_fc }

http-response set-header Strict-Transport-Security “max-age=16000000; includeSubDomains; preload;”

acl host_app1 hdr_sub(host) -i app1.example.com

acl host_app2 hdr_sub(host) -i app2.example.com

use_backend be_app1 if host_app1

use_backend be_app2 if host_app2

backend be_app1

    mode http

    log global

    balance source

    option httpchk

    http-request add-header X-Forwarded-Proto https if { ssl_fc }

    server  pp05 10.23.19.16:8484  ssl  check verify none  fall 3 rise 2

    server  pp06 10.23.19.17:8484  ssl  check verify none  fall 3 rise 2

    http-request set-header X-Forwarded-Port %[dst_port]

    http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend be_app2

    mode http

    log global

    balance source

    option httpchk

    http-request add-header X-Forwarded-Proto https if { ssl_fc }

    server pp03  10.29.18.18:8388  ssl check verify none fall 3 rise 2

    server pp04  10.29.19.16:8388  ssl check verify none fall 3 rise 2

    http-request set-header X-Forwarded-Port %[dst_port]

    http-request add-header X-Forwarded-Proto https if { ssl_fc }

I can confirm the above configuration works both browser and api calls. WIll there be any issues using wild card certificates?Do i need to remove or add any settings from the the above config ?

Thanks,
Sri

I don’t see anything wrong with it. Terminating SSL on haproxy and accessing the the host header is the better way in this case indeed.

There is no issue with wildcard certificates in this case (the issue is with overlapping certificates, often wildcard certificates between different backends while SNI routing, because browser will reuse wrong sessions but that’s not the case here).

I noticed you are using hdr_sub. For exact matches you can just use hdr.

It works great.I replaced hdr_sub to hdr.

Thanks again for your help.

-Sri