Scaling HAProxy to 100.000 SSL certificates

Help!
1

Hi !

I’ve been trying to load 100.000 distinct SSL certificates with HAProxy v1.8.13, to no avail.

My test server is a virtual machine with 4 vCPU and 8GB of RAM.

Here’s my configuration so far :

global
  daemon
  user haproxy
  group haproxy
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  stats maxconn 10
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/haproxy
  tune.ssl.default-dh-param 2048
  ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
  ssl-default-bind-options no-sslv3 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
  ssl-default-server-options no-sslv3 no-tls-tickets
  maxconn 100000
  spread-checks 4
  nbthread 4

defaults
  log global
  option httplog
  option dontlognull
  maxconn 100000
  timeout client 60s
  timeout server 60s
  timeout queue 60s
  timeout connect 4s
  timeout http-request 5s
  option httpclose
  option abortonclose
  option forwardfor
  option httpchk GET / HTTP/1.0\r\nUser-agent:\ haproxy-httpchk
  retries 2
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend ft_frontend
  mode http
  bind 127.0.0.1:80
  bind ::1:80
  acl servers_down nbsrv(bk_backend) lt 1
  monitor-uri /ping
  monitor fail if servers_down
  option http-server-close
  default_backend bk_backend

frontend ft_frontends
  mode http
  bind 127.0.0.1:443 ssl crt /etc/ssl/haproxy
  bind ::1:443 ssl crt /etc/ssl/haproxy
  acl servers_down nbsrv(bk_backend) lt 1
  monitor-uri /ping
  monitor fail if servers_down
  option http-server-close
  default_backend bk_backend

backend bk_backend
  mode http
  balance roundrobin
  server www1 10.0.0.1:8080 check inter 5s maxconn 25000
  server www2 10.0.0.1:8081 check inter 5s maxconn 25000
  server www3 10.0.0.2:8080 check inter 5s maxconn 25000
  server www4 10.0.0.2:8081 check inter 5s maxconn 25000

I’ve generated 100.000 self-signed SSL certificates using EasyRSA (2048 bits RSA), all of them stored in the /etc/ssl/haproxy folder.

–

Here are the result I am seeing when I reload HAProxy :

  • with 10.000 certificates
time service haproxy reload
[ ok ] Reloading haproxy: haproxy.
service haproxy reload  49,56s user 6,02s system 97% cpu 56,987 total
  • with 20.000 certificates
time service haproxy reload
[ ok ] Reloading haproxy: haproxy.
service haproxy reload  94,24s user 9,59s system 99% cpu 1:44,11 total
  • with 30.000 certificates
time service haproxy reload
[ ok ] Reloading haproxy: haproxy.
service haproxy reload  143,05s user 14,41s system 99% cpu 2:37,94 total
  • with 40.000 certificates
time service haproxy reload
[ ok ] Reloading haproxy: haproxy.
service haproxy reload  190,40s user 23,97s system 96% cpu 3:41,17 total
  • with 50.000 certificates

The process gets killed with an out of memory error.

–

The reload process seems to be pretty CPU heavy (100% usage during the whole reload process), but to only use one of the available vCPUs.

It would also seem HAProxy takes 5 more seconds to reload per 1.000 certificates I add. Is there any way to speed up the reload process ? I have not found anything regarding this in the documentation.

I’m also suprised to get an out of memory issue while loading ~400MB certificates. Am I missing something obvious ?

I also had a benchmark running during the reloads (1000 request per seconds towards the HAProxy internal monitoring URL I configured on /ping). No requests were lost during any of my tests, which is quite impressive !

Any help or tips would be appreciated !

Thanks !

2

8GB RAM is certainly completely undersized for this.

Correct, reloading means you need twice the RAM (the old process is still running and handling traffic while the new process loads the certificates, etc). If you don’t have enough, the process will get OOM killed.

That’s right. Loading thousands of certificates is going to require CPU power and will hit a single core.

I don’t see how this could be improved.

The size of the certificates on disk have very little to do with the memory usage.

I’m not suggesting there isn’t room for improvement; I’m not an expert in openssl or haproxy internals, however it looks to me like your expectations are far away from reality, so my question is: are you using a different product/project where the RAM usage is so much lower than with haproxy? I would like to understand where those expectations are coming from (if there is a more scalable way to do things, I would certainly like to know it).

3

Thanks for your reply !

Sorry, English is not my first language, so I might not have formulated my post properly.

I am not using any other product, no. I am just wondering if there’s any way to optimize the configuration I’ve come up with so far to obtain better results using my current server.

I guess I also should check if this scales linearly and if the process gets killed with 25.000 certificates on a 4GB machine, and with 100.000 certificates on a 16GB machine.

4

I doubt there is.

That’s what I’d expect, yes.