Hi guys,
I’d appreciate if anyone can give me couple of suggestions for the issue I have with SSL.
I know that sounds like certificate issue, but it happens only when I have big spike of new connections.
I am running haproxy 1.5.14 on Azure and using SSL termination.
Haproxy works perfectly well when load rises gradually, but everything goes bad if I have instant load.
In normal situation qmax goes up to 3000 and per thread and cpu core is loaded not higher than 75%.
So if I restart haproxy during daily load, haproxy might fill CPU usage up to 100% and be unable to handle more than 700-800 requests per thread.
When it comes to that limit, I see rate of new requests lowers down to 2-5
Haproxy log become mostly filled with tls/1: SSL handshake failure errors.
If I add more haproxy instances into balance, it becomes normal.
I don’t have issues with entropy:
cat /proc/sys/kernel/random/entropy_avail
885
I tried to add conneciton rate limits:
maxsessrate 100
maxsslrate 100
maxconnrate 100
that had no effect. Everything stops at about 800 connections and then whole log filled with SSL handshake failures.
I tried to play around with timeouts
changed timeout connect as:
- 500
- 50000
- 30s
No effect
Can anyone suggest anything here? I have no idea how to debug that.
Here is the config file I use:
global log /dev/log local0 log /dev/log local1 notice stats socket /var/run/haproxy.p1.sock mode 660 group nagios level admin process 1 stats socket /var/run/haproxy.p2.sock mode 600 level admin process 2 stats socket /var/run/haproxy.p3.sock mode 600 level admin process 3 stats socket /var/run/haproxy.p4.sock mode 600 level admin process 4 stats timeout 2m #Wait up to 2 minutes for input chroot /var/lib/haproxy user haproxy group haproxy daemon nbproc 4 cpu-map 1 0 # first arg is process number (1-based); second arg is cpu number (0-based) cpu-map 2 1 cpu-map 3 2 cpu-map 4 3
# SSL/TLS settings ca-base /etc/ssl/certs crt-base /etc/ssl/private tune.ssl.default-dh-param 2048 tune.ssl.cachesize 10000000 tune.ssl.lifetime 86400 #tune.ssl.maxrecord 2859 tune.ssl.maxrecord 1400 # TCP window size ssl-default-bind-options no-sslv3 no-tls-tickets ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
maxconn 60000 maxsslconn 60000 # maxsessrate 100 # maxsslrate 100 # maxconnrate 100
defaults log global option dontlognull option dontlog-normal timeout connect 5000 timeout client 50000 timeout server 50000 bind-process all # not needed, but worthwhile being explicit
listen stats bind :2100 process 1 bind :2101 process 2 bind :2102 process 3 bind :2103 process 4 mode http log global stats enable stats realm stats_process stats uri / stats refresh 15s stats show-legends stats show-node stats auth xxxxxxxxxxxxx
frontend tls mode tcp maxconn 60000 option tcplog bind *:443 ssl crt-list /etc/ssl/private/certificates.txt npn http/1.1 default_backend frontend_service
backend frontend_service mode tcp option tcplog option httpchk GET /status fullconn 60000
# 2 second 'inter'val between health checks. 2 failures to remove a server. 2 successes to add it back default-server inter 8s fall 2 rise 2 timeout check 8s
balance leastconn
server SRV1 SRV1:80 maxconn 2000 check port 3000 .... server SRV60 SRV1:80 maxconn 2000 check port 3000
Thank you!
Pavel