Using SSL with HAProxy for docker containers


#1

Hi, I am currently using HAProxy to split web traffic between my docker sites, and all other sites. I’d now like to use SSL for my sites.

EDIT:

For the purpose of those coming across this thread in future I have summarised what I have learnt as follows:

  • It’s easier than you think!
  • You don’t need to worry whether your sites are served via Docker, or Apache - it’s HAProxy that speaks to the browser so you can let HAProxy handle it all.
  • You simply need to add bind *:443 ssl crt /etc/haproxy/certs/ to your front-end then just make sure your certs are in that directory.
  • See this article on how to set-up and renew Lets Encrypt certs.

That’s it!

Huge thanks to @lukastribus for all his help :slight_smile:

/////////End Edit/////

Here is my current cfg file:

global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    # log         127.0.0.1 local2

    # chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    # option forwardfor       except 127.0.0.0/8
    option forwardfor
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000


frontend http-in
        bind *:80
        default_backend main_apache_sites

        # Define hosts
	      redirect prefix http://discourse-forum-1.com code 301 if { hdr(host) -i www.discourse-forum-1.com }
        acl host_discourse hdr(host) -i discourse-forum-1.com
	      redirect prefix http://discourse-forum-2.com code 301 if { hdr(host) -i www.discourse-forum-2.com }
        acl host_discourse_2 hdr(host) -i discourse-forum-2.com		
	      redirect prefix http://discourse-forum-3.com code 301 if { hdr(host) -i www.discourse-forum-3.com }
        acl host_discourse_3 hdr(host) -i discourse-forum-3.com
		
        # which one to use
        use_backend discourse_docker if host_discourse
        use_backend discourse_docker_2 if host_discourse_2
        use_backend discourse_docker_3 if host_discourse_3	


backend main_apache_sites
    	server server1 127.0.0.1:8080 cookie A check
	    cookie JSESSIONID prefix nocache

backend discourse_docker
    	server server2 127.0.0.1:8888 cookie A check
	    cookie JSESSIONID prefix nocache

backend discourse_docker_2
    	server server2 127.0.0.1:8889 cookie A check
	    cookie JSESSIONID prefix nocache

backend discourse_docker_3
    	server server2 127.0.0.1:8890 cookie A check
	    cookie JSESSIONID prefix no cache

If you look at # Define hosts you’ll see my docker sites (which are actually Discourse forums - the same as what you are running here). I’d like to enable SSL by using their letsencrypt plugin (if possible) here are the details but essentially it take cares of the certificate and auto-renews it.

Does anyone know what my haproxy.cfg file should look like if I wanted to enable SSL as per their guide?

Thanks in advance.


#2

Anyone have any ideas on this please?


#3

Anyone know why all of my sites become inaccessible after adding this to the config?

frontend https-in
        bind *:443 ssl crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
        default_backend main_apache_ssl_sites
        
backend main_apache_ssl_sites
    	server server4 127.0.0.1:4433 cookie A check ssl verify none
	    cookie JSESSIONID prefix no cache

#4

Good timing. I’ve been working on the same thing. Check out:


#5

Also here’s a sample haproxy.cfg: https://gist.github.com/bradjones1/ed6d01e311e89b6383e4b1625e72c64a


#6

What are you trying to accomplish? Why would you encrypt your backend/loopback traffic? Is 4433 a SSL enabled port on 127.0.0.1?


#7

Thanks for that @bradj I will take a look.

@lukastribus, I am trying to enable https on some of the sites on the server (not all of them). The original setup was the usual Apache listening on port 80. When I installed HAProxy I configured Apache to listen on port 8080 and that’s what’s used for the default_backend. HAProxy sends requests to one of three Docker containers if one of those domains is requested, otherwise it sends everything else to the default backend. All this works fine for normal http requests.

What I’m trying to do is enable https on my docker containers (which are actually Discourse forums) as well as some of my normal ‘default_backend’ sites.

I have actually been able to enable https successfully on some of the Apache default_backend sites (I guess because Apache is listing on port 443 as normal and HAProxy is just ignoring that port since I don’t tell it otherwise). Hence port 4433, which is what I was going to change Apache’s ssl port to.

Does that better explain what I am trying to do? Thanks for your help - I really appreciate it.


#8

Ok, but what you want to do is enabled SSL between the client/browser to haproxy. There is no need to enable SSL between haproxy and the apache backend, or is it?

If you enable SSL both on the frontend and the backend, your server’s CPU load due to SSL is multiplied by a factor of 3 (SSL termination on the haproxy frontend vs SSL termination on the haproxy frontend + backend + apache backend).


#9

How would I do that Lukas? (Sorry if that’s a dumb question)


#10

You keep using your non-SSL backend “main_apache_sites” without modifying it. No need for the main_apache_ssl_sites backend.

You could even do it without the dedicated ssl frontend, its not needed.

So you basically just add an additional bind line to your existing frontend:

frontend http-in
 bind *:80
 bind *:443 ssl crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
 default_backend main_apache_sites
backend main_apache_sites
 server server1 127.0.0.1:8080 cookie A check cookie JSESSIONID prefix nocache

#11

Thanks Lukas. Unfortunately when I try that all of my sites become inaccessible. Even simply changing:

bind *:80

to:

bind *:80,:443

and all of the sites become inaccessible. Could I have something else incorrectly configured in my cfg file (which is as per the first post).


#12

Don’t do this, this configuration is wrong. You want SSL on port 443 and plaintext HTTP on port 80, you cannot configure it on the same bind line.

I don’t see why listening to port 443 would make your sites inaccessible. Does haproxy even start correctly? Can it bind to port 443 (or is there another apache instance listening on port 443)? Do you see anything relevant on stdout/stderr or in the syslog? What about haproxy logging?


#13

Ok I’ve changed Apache’s SSL config to listen on port 4433 (and made sure none of the existing records refer to 443), however the sites are still inaccessible with the following uncommented:

bind *:443 ssl crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

However simply adding:

bind *:443

Does not render the sites inaccessible.

I had thought that /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem was where all .pems were getting concatenated, but perhaps that’s not the case (I am using Webmin’s letsencrypt client to create the certs and even adding /etc/webmin/webmin/letsencrypt.pem as the location renders the sites inaccessible).

If I am using HAProxy, do I still need to get a cert for every domain? Or just one for HAProxy as that’s what the browser connects to?

Should I be looking at something like this? https://www.digitalocean.com/community/tutorials/how-to-secure-haproxy-with-let-s-encrypt-on-centos-7

Ideally, I could just use Webmin’s Letsencrypt client as it also handles renewals for all my Apache sites, and for the Docker containers, just use the Discourse scrip per container as per my first post - but I’m guessing this is not possible?

Any advice you can give will be greatly appreciated. And I’m sorry that I keep replying with more questions.


#14

Don’t jump to conclusions. Respond to the existing question above, like when the sites are inaccessible, is haproxy even correctly started or does it fail to start? Start it manually and check if there are any error messages or warnings.

Also provide the output of “haproxy -vv”.


#15

Sorry I forgot to answer you questions (I couldn’t find the log file anywhere).

Running haproxy -vv didn’t work for me so I tried systemctl status haproxy and got:

systemd[1]: Started HAProxy Load Balancer.
systemd[1]: Starting HAProxy Load Balancer...
haproxy-systemd-wrapper[19941]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
haproxy-systemd-wrapper[19941]: [ALERT] 159/134514 (19942) : parsing [/etc/haproxy/haproxy.cfg:49] : 'bind *:443' : unable to load SSL certificate from PEM file '/etc/webmin/webmin/letsencrypt.pem'.
haproxy-systemd-wrapper[19941]: [ALERT] 159/134514 (19942) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
haproxy-systemd-wrapper[19941]: [WARNING] 159/134514 (19942) : config : log format ignored for frontend 'http-in' since it has no log address.
haproxy-systemd-wrapper[19941]: [ALERT] 159/134514 (19942) : Proxy 'http-in': no SSL certificate specified for bind '*:443' at [/etc/haproxy/haproxy.cfg:49] (use 'crt').
haproxy-systemd-wrapper[19941]: [ALERT] 159/134514 (19942) : Fatal errors found in configuration.
haproxy-systemd-wrapper[19941]: haproxy-systemd-wrapper: exit, haproxy RC=256
```

Looks like the PEM file is not right. I've looked at the files created by the webmin lets encrypt script and it looks like it creates 3 files in the root directory for the domain:

```
-rwxr-xr-x   1 domain.com domain.com 1647 May 23 23:49 ssl.ca*
-rwx------   1 domain.com domain.com 2159 May 23 23:49 ssl.cert*
-rwx------   1 domain.com domain.com 3243 May 23 23:49 ssl.key*

```


I can't see any related .PEM files.

```
locate *.pem*
/etc/dhparams.pem
/etc/pki/ca-trust/extracted/pem/email-ca-bundle.pem
/etc/pki/ca-trust/extracted/pem/objsign-ca-bundle.pem
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
/etc/pki/dovecot/certs/dovecot.pem
/etc/pki/dovecot/private/dovecot.pem
/etc/pki/tls/cert.pem
/etc/postfix/postfix.cert.pem
/etc/postfix/postfix.key.pem
/etc/usermin/miniserv.pem
/etc/webmin/miniserv.pem
/etc/webmin/webmin/letsencrypt.pem
```

Any ideas what to try next?

#16

What I suspected: haproxy doesn’t even start because of fatal errors.

Make sure /etc/webmin/webmin/letsencrypt.pem is accessible by haproxy and contains the correct certificate.


#17

Which certificate(/s) does it need to contain Lukas?

Several domains will be moving to https, do I just add each ssl.cert (i.e. one of the files generated by letsencrypt/webmin) into one .pem file?


#18

The pem file must contain the the private key, the certificate and the intermediate certificate [1]. If you have more than one certificate you need to use multiple files and point to each of them or the directory containing all those files.

[1] http://www.jebriggs.com/blog/2015/01/haproxy-and-ssl-sni-support/


#19

Great! I can now get the sites to resolve :slight_smile:

I am unsure about a few things though…

1 - The link says I need to create a single PEM file with key, crt and chain entries . I do not know what chain entries are. The only three files the webmin lets encrypt client seems to produce are ssl.key, ssl.cert and ssl.ca. I have tried concatenating the cert and key, as well as all three and both versions work. What is the correct approach here?

2 - Can I create one single .pem file that contains all of the separate .pems? Or do I need to specify each file separately (or a directory)

3 - Are there any recommended lets encrypt scripts to use with HAProxy? Having to manually renew the pems (as certs expire every two months) could be a nuisance.

Any other advice appreciated (and thanks once again for all your help).


#20

The chain entry would be ssl.ca. So the concatenation would look like this:
cat ssl.key ssl.cert ssl.ca >ssl.pem

Like I said its one key->cert combination per file only, and you can either specify multiple files or a directory.

Take a look at this: https://github.com/janeczku/haproxy-acme-validation-plugin