Apache multiple SSL VirtualHost single IP and port

I’ve been struggling with something, and want to make sure I’m not missing something simple. I don’t think this can be done, but would like confirmation.

I have one Apache server with multiple VirtualHost configs:

<VirtualHost *:443>
    ServerName api-test-haproxy.neatoserver.lan

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/api-test-haproxy.neatoserver.lan.crt
    SSLCertificateKeyFile /etc/pki/tls/private/api-test-haproxy.neatoserver.lan.key

    DocumentRoot /var/www/api-test-haproxy.neatoserver.lan
    <Directory /var/www/api-test-haproxy.neatoserver.lan>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>

<VirtualHost *:443>
    ServerName api2-test-haproxy.neatoserver.lan

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/api2-test-haproxy.neatoserver.lan.crt
    SSLCertificateKeyFile /etc/pki/tls/private/api2-test-haproxy.neatoserver.lan.key

    DocumentRoot /var/www/api2-test-haproxy.neatoserver.lan
    <Directory /var/www/api2-test-haproxy.neatoserver.lan>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>

Going to https://api-test-haproxy.neatoserver.lan shows the proper api-test site and files, and going to https://api2-test-haproxy.neatoserver.lan shows the other site and files. All good on the Apache side of things.

Now I want to bring HAProxy into the mix, and get another server for HA-ing the sites. Update my DNS names to point the CNAME to the HAProxy VIP, and configure HAProxy like so:

defaults
    mode                    http
    balance                 source
    log                     global
    option                  httplog

frontend front_https
    bind *:443 ssl crt /etc/haproxy/certs/
    option forwardfor except 127.0.0.0/8
    use_backend back_api if { ssl_fc_sni api-test.neatoserver.lan }
    use_backend back_api2 if { ssl_fc_sni api2-test.neatoserver.lan }

backend back_api
    server api-01 api-01.neatoserver.lan:443 check ssl verify none
    server api-02 api-02.neatoserver.lan:443 backup check ssl verify none

backend back_api2
    server api2-01 api-01.neatoserver.lan:443 check ssl verify none
    server api2-02 api-02.neatoserver.lan:443 backup check ssl verify none

Going to https://api-test-haproxy.neatoserver.lan still shows the proper api-test site and files for api, but now going to https://api2-test-haproxy.neatoserver.lan is broken and incorrectly shows the sites and files for api-test-haproxy.neatoserver.lan. It would appear that HAProxy doesn’t pass SNI to the proper VirtualHost.

Only option I’ve found to get this to work through HAProxy, is defining a separate port for the VirtualHost(s) config. Like this:

Listen 8443
<VirtualHost *:8443>
    ServerName api-test-haproxy.neatoserver.lan

Listen 8444
<VirtualHost *:8444>
    ServerName api2-test-haproxy.neatoserver.lan

And then updating HAProxy to use the separate ports for the specified sites:

backend back_api
    server api-01 api-01.neatoserver.lan:8443 check ssl verify none
    server api-02 api-02.neatoserver.lan:8443 backup check ssl verify none

backend back_api2
    server api2-01 api-01.neatoserver.lan:8444 check ssl verify none
    server api2-02 api-02.neatoserver.lan:8444 backup check ssl verify none

Is this correct? Is this the only way to get HAProxy to work with multiple VirtualHost on the same server?

Thanks!
Danny

you have to pass the domain to your backend servers.

in my config i have

    capture request header Host len 32
    capture request header User-Agent len 64
    http-request set-header X-SSL                       %[ssl_fc]
    http-request set-header X-SSL-HOST                  %[ssl_fc_sni]
    http-request set-header X-Forwarded-Proto https

i think the capture lines are important. i also set some additonal headers, to have it in the cgi environment.

without the capture host apache will use the first SSL-Virtual Host it can find if there is no SNI (host) available (or if the host don’t macht to any virtual hosts)

markus

OK, I must be missing something. I’ve updated my config like so:

defaults
    mode                    http
    balance                 source
    log                     global
    option                  httplog

frontend front_https
    bind *:443 ssl crt /etc/haproxy/certs/
    option forwardfor except 127.0.0.0/8

    capture request header Referer len 64
    capture request header Content-Length len 10
    capture request header Host len 32
    capture request header User-Agent len 64

    http-request set-header X-SSL                       %[ssl_fc]
    http-request set-header X-SSL-HOST                  %[ssl_fc_sni]
    http-request set-header X-Forwarded-Proto https

    use_backend back_api if { ssl_fc_sni api-test-haproxy.neatoserver.lan }
    use_backend back_api2 if { ssl_fc_sni api2-test-haproxy.neatoserver.lan }

backend back_api
    server api-01 api-01.neatoserver.lan:443 check ssl verify none
    server api-02 api-02.neatoserver.lan:443 backup check ssl verify none

backend back_api2
    server api2-01 api-01.neatoserver.lan:443 check ssl verify none
    server api2-02 api-02.neatoserver.lan:443 backup check ssl verify none

…restarted everything, but the issue remains. This is on a CentOS Stream server running: haproxy-1.8.27-5.el8.x86_64

Anything other suggestions?

Quickly looking on your HAProxy configuration.
It seems you are defining the same servers in both backends and api-02 is backup for both.
It means that it is always routed to api-01 if it is healthy.

backend back_api
    server api-01 api-01.neatoserver.lan:443 check ssl verify none
    server api-02 api-02.neatoserver.lan:443 backup check ssl verify none

backend back_api2
    server api2-01 api-01.neatoserver.lan:443 check ssl verify none
    server api2-02 api-02.neatoserver.lan:443 backup check ssl verify none

Also I noticed that you aren’t using any HTTP request modification so you could consider TCP mode.
See this blog post for more info.

BR

Yes, that is correct. Two identical servers, hosting multiple VirtualHost configs, one is primary, the other is backup.

I’d rather keep SSL termination at the HAProxy, so that is why I’m using HTTP mode. Although, I did try with TCP mode, and it made no difference.

Can you share the logs from HAProxy and Apache pls?
There should be info which backend was used.
Not sure if host header will be present.

@findmyname apologies, I’m not following your train of thought. The HAProxy frontend config:

    use_backend back_api if { ssl_fc_sni api-test-haproxy.neatoserver.lan }
    use_backend back_api2 if { ssl_fc_sni api2-test-haproxy.neatoserver.lan }

…is doing it’s job and pushing the correct ssl_fc_sni to the correct backend:

0000000d:front_https.clireq[001e:ffffffff]: GET / HTTP/1.1
0000000d:front_https.clihdr[001e:ffffffff]: Host: api-test-haproxy.neatoserver.lan
0000000d:front_https.clihdr[001e:ffffffff]: User-Agent: curl/7.69.1
0000000d:front_https.clihdr[001e:ffffffff]: Accept: */*
0000000d:back_api.srvrep[001e:0020]: HTTP/1.1 200 OK
0000000d:back_api.srvhdr[001e:0020]: Date: Tue, 22 Nov 2022 18:10:27 GMT
0000000d:back_api.srvhdr[001e:0020]: Server: Apache
0000000d:back_api.srvhdr[001e:0020]: Strict-Transport-Security: max-age=31536000; includeSubDomains
0000000d:back_api.srvhdr[001e:0020]: X-Frame-Options: SAMEORIGIN
0000000d:back_api.srvhdr[001e:0020]: Last-Modified: Tue, 15 Nov 2022 18:11:31 GMT
0000000d:back_api.srvhdr[001e:0020]: Accept-Ranges: bytes
0000000d:back_api.srvhdr[001e:0020]: Content-Length: 33
0000000d:back_api.srvhdr[001e:0020]: X-XSS-Protection: 1; mode=block
0000000d:back_api.srvhdr[001e:0020]: X-Content-Type-Options: nosniff
0000000d:back_api.srvhdr[001e:0020]: Content-Type: text/html; charset=UTF-8
00000010:front_https.clireq[001e:ffffffff]: GET / HTTP/1.1
00000010:front_https.clihdr[001e:ffffffff]: Host: api2-test-haproxy.neatoserver.lan
00000010:front_https.clihdr[001e:ffffffff]: User-Agent: curl/7.69.1
00000010:front_https.clihdr[001e:ffffffff]: Accept: */*
00000010:back_api2.srvrep[001e:001f]: HTTP/1.1 200 OK
00000010:back_api2.srvhdr[001e:001f]: Date: Tue, 22 Nov 2022 18:11:58 GMT
00000010:back_api2.srvhdr[001e:001f]: Server: Apache
00000010:back_api2.srvhdr[001e:001f]: Strict-Transport-Security: max-age=31536000; includeSubDomains
00000010:back_api2.srvhdr[001e:001f]: X-Frame-Options: SAMEORIGIN
00000010:back_api2.srvhdr[001e:001f]: Last-Modified: Tue, 15 Nov 2022 18:11:31 GMT
00000010:back_api2.srvhdr[001e:001f]: Accept-Ranges: bytes
00000010:back_api2.srvhdr[001e:001f]: Content-Length: 33
00000010:back_api2.srvhdr[001e:001f]: X-XSS-Protection: 1; mode=block
00000010:back_api2.srvhdr[001e:001f]: X-Content-Type-Options: nosniff
00000010:back_api2.srvhdr[001e:001f]: Content-Type: text/html; charset=UTF-8

That’s not really the issue.

I think @Markus is more on the right track, as HAProxy isn’t sending over the correct SNI information, so Apache is getting confused and using the default VirtualHost, because it is not seeing the proper request. Unfortunately, even with that updated “catpure request” config, still no go for me.

Is it ok that your Apache config contains different server names ?

ServerName api-test-haproxy.neatoserver.lan

As far my understanding goes it should be equal to host header right ?

Host: api-test.neatoserver.lan

Yes, that is how you achieve name-based virtual hosting: Name-based Virtual Host Support - Apache HTTP Server Version 2.4

Also, as I mentioned above, Apache configs are working as expected. If I remove HAProxy from the situation (update DNS to point back directly at the Apache servers) everything works exactly how you would expect:

Sorry I’m kinda confused here. In the example above you are testing different FQDN https://api-test-haproxy.neatoserver.lan but the logs contains api-test.neatoserver.lan. Could you test also behaviour without HAProxy with hostname api-test.neatoserver.lan ?

E.g. HAProxy will not change request host header. You need to configure it if that is required.

Apologies, that is just a typo. I’m rewriting the configs on the fly for this post, as I’m obviously not wanting to use the actual names of our servers. The above HAProxy should read:

frontend front_https
    bind *:443 ssl crt /etc/haproxy/certs/
    option forwardfor except 127.0.0.0/8
    use_backend back_api if { ssl_fc_sni api-test-haproxy.neatoserver.lan }
    use_backend back_api2 if { ssl_fc_sni api2-test-haproxy.neatoserver.lan }

Nice catch, and sorry for the confusion. I can’t edit the original post, so future readers are going to be even more confused :slight_smile:

Either way, that’s what it should look like, but is not working.

For clarity, here is the current Apache and HAProxy config that is NOT working:

<VirtualHost *:443>
    ServerName api-test-haproxy.neatoserver.lan

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/api-test-haproxy.neatoserver.lan.crt
    SSLCertificateKeyFile /etc/pki/tls/private/api-test-haproxy.neatoserver.lan.key

    DocumentRoot /var/www/api-test-haproxy.neatoserver.lan
    <Directory /var/www/api-test-haproxy.neatoserver.lan>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>

<VirtualHost *:443>
    ServerName api2-test-haproxy.neatoserver.lan

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/api2-test-haproxy.neatoserver.lan.crt
    SSLCertificateKeyFile /etc/pki/tls/private/api2-test-haproxy.neatoserver.lan.key

    DocumentRoot /var/www/api2-test-haproxy.neatoserver.lan
    <Directory /var/www/api2-test-haproxy.neatoserver.lan>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>
frontend front_https
    bind *:443 ssl crt /etc/haproxy/certs/
    option forwardfor except 127.0.0.0/8

    capture request header Referer len 64
    capture request header Content-Length len 10
    capture request header Host len 32
    capture request header User-Agent len 64

    http-request set-header X-SSL                       %[ssl_fc]
    http-request set-header X-SSL-HOST                  %[ssl_fc_sni]
    http-request set-header X-Forwarded-Proto https

    use_backend back_api if { ssl_fc_sni api-test-haproxy.neatoserver.lan }
    use_backend back_api2 if { ssl_fc_sni api2-test-haproxy.neatoserver.lan }

backend back_api
    server api-01 api-01.neatoserver.lan:443 check ssl verify none
    server api-02 api-02.neatoserver.lan:443 backup check ssl verify none

backend back_api2
    server api2-01 api-01.neatoserver.lan:443 check ssl verify none
    server api2-02 api-02.neatoserver.lan:443 backup check ssl verify none

Never use ssl_fc_sni if you can use hdr(hosts).

On the backend servers, specify SNI for health checks and traffic:

sni str(api-test-haproxy.neatoserver.lan) check-sni api-test-haproxy.neatoserver.lan

so that Apache SSL Vhost can actually match something.

1 Like

Understood, and thanks. Updated the frontends to:

    use_backend back_api if { hdr(host) -i api-test-haproxy.neatoserver.lan }
    use_backend back_api2 if { hdr(host) -i api2-test-haproxy.neatoserver.lan }

Updated the backends to:

backend back_api
    server api-01 api-01.neatoserver.lan:443 check ssl verify none sni str(api-test-haproxy.neatoserver.lan) check-sni api-test-haproxy.neatoserver.lan
    server api-02 api-02.neatoserver.lan:443 backup check ssl verify none sni str(api-test-haproxy.neatoserver.lan) check-sni api-test-haproxy.neatoserver.lan

backend back_api2
    server api2-01 api-01.neatoserver.lan:443 check ssl verify none sni str(api2-test-haproxy.neatoserver.lan) check-sni api2-test-haproxy.neatoserver.lan
    server api2-02 api-02.neatoserver.lan:443 backup check ssl verify none sni str(api2-test-haproxy.neatoserver.lan) check-sni api2-test-haproxy.neatoserver.lan

Unfortunately, the issue remains.

Not really sure why are you using HTTPS between haproxy and apache when you want to terminate SSL on haproxy, but I guess there is a reason for it.

Stop haproxy completely, make sure no leftover processes are running, and start haproxy again.

Sometimes when testing different configurations and reloading/restarting a lot, old processes remain and keep terminating even new connections, due to kernel load balancing. So make sure that is not the case here.

I’m already making sure I stop/start properly by not using anything like Systemd, and running in debug mode:
sudo haproxy -f /etc/haproxy/haproxy.cfg -d

In between each update/test, I Ctrl+c the above, and re-run. No other process is running.

Not sure what’s happening.

Check haproxy logs, are you you actually routed through the second, api2 backend or are you always routed through the first?

I suggest you capture the SSL handshake between haproxy and the backend server to see if the SNI that haproxy sends is correct or not.

Provide both haproxy and apache logs of the a good (api) and bad (api2) request.

1 Like

@lukastribus

Thank you so much for your help! Your solution with the “specify SNI for health checks and traffic” was the proper fix. The only reason it wasn’t initially working was because I had a completely unrelated Apache config breaking things. So, in the end, here are the proper configs.

Apache:

<VirtualHost *:443>
    ServerName api-test-haproxy.neatoserver.lan

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/api-test-haproxy.neatoserver.lan.crt
    SSLCertificateKeyFile /etc/pki/tls/private/api-test-haproxy.neatoserver.lan.key

    DocumentRoot /var/www/api-test-haproxy.neatoserver.lan
    <Directory /var/www/api-test-haproxy.neatoserver.lan>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>

<VirtualHost *:443>
    ServerName api2-test-haproxy.neatoserver.lan

    SSLEngine on
    SSLCertificateFile /etc/pki/tls/certs/api2-test-haproxy.neatoserver.lan.crt
    SSLCertificateKeyFile /etc/pki/tls/private/api2-test-haproxy.neatoserver.lan.key

    DocumentRoot /var/www/api2-test-haproxy.neatoserver.lan
    <Directory /var/www/api2-test-haproxy.neatoserver.lan>
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
    </Directory>
</VirtualHost>

HAProxy:

frontend front_https
    bind *:443 ssl crt /etc/haproxy/certs/
    option forwardfor except 127.0.0.0/8

    # ACLs
    use_backend back_api if { hdr(host) -i api-test-haproxy.neatoserver.lan }
    use_backend back_api2 if { hdr(host) -i api2-test-haproxy.neatoserver.lan }

backend back_api
    server api-01 api-01.neatoserver.lan:443 check ssl verify none sni str(api-test-haproxy.neatoserver.lan) check-sni api-test-haproxy.neatoserver.lan
    server api-02 api-02.neatoserver.lan:443 backup check ssl verify none sni str(api-test-haproxy.neatoserver.lan) check-sni api-test-haproxy.neatoserver.lan

backend back_api2
    server api2-01 api-01.neatoserver.lan:443 check ssl verify none sni str(api2-test-haproxy.neatoserver.lan) check-sni api2-test-haproxy.neatoserver.lan
    server api2-02 api-02.neatoserver.lan:443 backup check ssl verify none sni str(api2-test-haproxy.neatoserver.lan) check-sni api2-test-haproxy.neatoserver.lan

1 Like