High CPU Usage with squid serverfarm

Help!
1

Hi all,

Since we migrate our squid on Haproxy we experiencing high CPU usage on our HaProxy server. Theses Squid are used as proxy web for our users (the VIP of HAPROXY is directly set up on their browser)
We activate muthreading option on 4 vCPU but they are all at 90% at the top of the day.
I’ve also check that TCP ports are not exhausted.

Compilation option are the following : make TARGET=custom CPU=generic USE_OPENSSL=1 USE_ZLIB=1 USE_PCRE=1 USE_LINUX_SPLICE=1 USE_CPU_AFFINITY=1

We also used Keepalived for high availability in active/passive mode.<img

Can you help me please, we don’t have so many connections and I’m sure that HAPROXY can do better ?

You can find bellow the Hardware and Software configuration :

Vmware Esxi 5.5
Linux Red Hat 6.8
4 vCPU
4 Go RAM
20 Go Disk
3 Gigabits interfaces (One for the administration and the two other one for the VIP)

global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 2000000
chroot /product/haproxy
uid 99
gid 99
daemon
tune.ssl.default-dh-param 2048
nbproc 4
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3
#debug
#quiet

defaults
log global
option dontlognull
retries 3
option redispatch
maxconn 2000000
timeout connect 5000
timeout client 600000
timeout server 600000

    frontend stats
    bind  xxx.xxx.xxx.xxx:8080
    mode http
    stats enable
    stats uri /stats
    stats realm HAProxy Statistics
    stats auth admin:admin

################################
####### Ferme Proxy LAN ####
################################

frontend proxylan-rewrite
bind xxx.xxx.xxx.xxx:xx
mode http
option http-keep-alive
default_backend proxylan-backend

backend proxylan-backend
mode http
balance source
option prefer-last-server
option http-keep-alive
option forwardfor
server proxylan_1 xxx.xxx.xxx.xxx:xx check
server proxylan_2 xxx.xxx.xxx.xxx:xx check
server proxylan_3 xxx.xxx.xxx.xxx:xx check
server proxylan_4 xxx.xxx.xxx.xxx:xx check

2

You can also find bellow the http statistics :

Stats1.png1573×376 42.2 KB

3

Does the CPU load come directly from haproxy in userspace or is it in the system?

Do not compile with TARGET=custom, you should use TARGET=linux2628 for this setup.

Your maxconn number suggest you have at least 64GB RAM per process, with nbproc 4 that means you would need at least 256GB RAM. Maxconn 20000 is more realistic for this scenario.

Not sure what “Go RAM” means, but if it means ballooning, then don’t do it. Give the VM fixed resources, from a RAM and CPU perspective. Everything else will a have a horrible effect on performance.

And please do understand the limitation of the multiprocess mode. The stats interface displays informations of a single process (which you randomly hit, when you access it).

4

Hello,

Thank you for quick answer, by Go, I Mean GigaByte (french translation sorry), we have effectively 4 GB of RAM per server.

The CPU load is in the system only.

We will recompile the new version and set it up tomorow, I’ll keep you in touch.

Thank you again. !

5

Hello,

Good news, we compile HAPROXY with the new option this morning and the CPU usage decrease from 90% to 3%, many thanks for your help !!

Aurélien.

6

Great, thanks for the feedback.

Still, please fix your maxconn numbers adjusting it to your RAM, otherwise haproxy will consume to much RAM and get OOM-killed.

Read more about it here:
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4.2-maxconn