Hi All,
I have installed two CentOs 7 sevrers with KeepAlived and HAPROXY . (1.8.12-8a200c7, released 2018/06/27)
Keepalived works as expected.
HAPROXY works as expected.
But I have problem with performance. I tried to use apache benchmark to check how many request can server handle and I was little bit surprised. If tests are going over the HAPROXY test process is 3 times slower and there is a lot of 503.
If I am running test directly against web server it very fast without any errors.
Tests are running on third centos server in different VLAN.
AB directly:
CENTOS6 192.168.1.83 - 192.168.2.167
AB Ower HAPROXY:
CENTOS6 192.168.1.86 - 192.168.2.211 - 192.168.2.167
My config of HAPROXY is:
HAPROXY:
2CPU
4GB RAM.
2 NICs (1 DMZ, 1 LAN)
---------haproxy.cfg------------------------------------------------------------
global
log 127.0.0.1 local2
debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 20000
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 4096
# turn on stats unix socket
stats socket /var/lib/haproxy/stats mode 660 level admin
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 20000
frontend Default_FE
bind 172.16.1.201:80
bind 192.168.2.201:80
reqadd X-Forwarded-Proto:\ http
option http-tunnel
FRONTEND WEB SERVER
frontend FRONTEND-WEB
bind 172.16.1.211:80
bind 192.168.2.211:80
bind 172.16.1.211:443 ssl crt /cert/wild.pem
bind 192.168.2.211:443 ssl crt /cert/wild.pem
# option http-server-close
# option forwardfor except 127.0.0.0/8
# reqadd X-Forwarded-Proto:\ https
# reqadd X-Forwarded-Port:\ 443
rspadd Strict-Transport-Security:\ max-age=15768000
acl is_service-acc hdr(host) -i service.domain.com
use_backend service-acc if is_service-acc
backend service-acc
mode http
balance source
option forwardfor except 127.0.0.0/8
option http-server-close
option http-tunnel
option redispatch
balance roundrobin
stick-table type binary len 32 size 30k expire 30m
acl clienthello req_ssl_hello_type 1
acl serverhello rep_ssl_hello_type 2
tcp-request inspect-delay 5s
tcp-request content accept if clienthello
tcp-response content accept if serverhello
stick on payload_lv(43,1) if clienthello
stick store-response payload_lv(43,1) if serverhello
option httpchk GET /swagger/v1/swagger.json
cookie SERVERID insert indirect nocache
server WEB-A01 192.168.2.167:443 ssl verify none check cookie s1
HAPROXY LOGS:
Jul 23 15:45:38 localhost haproxy[1074]: 192.168.1.83:37794 [23/Jul/2018:15:45:33.822] FRONTEND service/WEB-A01 0/0/711/3877/4588 200 2875 - - ---- 9/9/8/7/0 0/0 “GET /service.svc HTTP/1.0”
Jul 23 15:45:38 localhost haproxy[1074]: 192.168.1.83:37848 [23/Jul/2018:15:45:33.859] FRONTEND service/WEB-A01 0/0/673/3877/4550 200 2875 - - ---- 8/8/7/7/0 0/0 “GET /service.svc HTTP/1.0”
Jul 23 15:45:38 localhost haproxy[1074]: 192.168.1.83:37842 [23/Jul/2018:15:45:33.859] FRONTEND service/WEB-A01 0/0/673/3878/4551 200 2875 - - ---- 7/7/6/6/0 0/0 “GET /service.svc HTTP/1.0”
Jul 23 15:45:38 localhost haproxy[1074]: 192.168.1.83:37806 [23/Jul/2018:15:45:33.822] FRONTEND service/WEB-A01 0/0/710/3878/4589 200 2875 - - ---- 6/6/5/4/0 0/0 “GET /service.svc HTTP/1.0”
Wrong:
Jul 23 15:45:34 localhost haproxy[1074]: 192.168.1.83:57804 [23/Jul/2018:15:45:33.008] FRONTEND-ACC service/WEB-A01 0/0/526/1003/1541 503 513 - - ---- 4397/4397/4396/4155/0 0/0 “GET /service.svc HTTP/1.0”
Jul 23 15:45:34 localhost haproxy[1074]: 192.168.1.83:57806 [23/Jul/2018:15:45:33.008] FRONTEND-ACC service/WEB-A01 0/0/526/1003/1541 503 513 - - ---- 4396/4396/4395/4155/0 0/0 “GET /service.svc HTTP/1.0”
Jul 23 15:45:34 localhost haproxy[1074]: 192.168.1.83:57802 [23/Jul/2018:15:45:33.008] FRONTEND-ACC service/WEB-A01 0/0/526/1003/1541 503 513 - - ---- 4395/4395/4394/4155/0 0/0 “GET /service.svc HTTP/1.0”
[root@haproxy-A01 ~]# ab -n 5000 -c 5000 -H “Host; service” 192.168.2.211
This is ApacheBench, Version 2.3 <Revision: 655654 >
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation,
Benchmarking 192.168.2.211 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests
Server Software: Microsoft-IIS/8.0
Server Hostname: 192.168.2.211
Server Port: 80
Document Path: /service.svc
Document Length: 2635 bytes
Concurrency Level: 5000
Time taken for tests: 3.486 seconds
Complete requests: 5000
Failed requests: 2883
(Connect: 0, Receive: 0, Length: 2883, Exceptions: 0)
Write errors: 0
Non-2xx responses: 2883
Total transferred: 7790354 bytes
HTML transferred: 6518153 bytes
Requests per second: 1434.48 [#/sec] (mean)
Time per request: 3485.574 [ms] (mean)
Time per request: 0.697 [ms] (mean, across all concurrent requests)
Transfer rate: 2182.64 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 84 149.8 57 1024
Processing: 136 1970 616.5 1853 3346
Waiting: 4 1964 619.2 1837 3346
Total: 137 2054 601.7 1922 3473
Percentage of the requests served within a certain time (ms)
50% 1922
66% 2098
75% 2327
80% 2550
90% 3052
95% 3252
98% 3357
99% 3383
100% 3473 (longest request)
Or
[root@haproxy-A01 ~]# ab -n 5000 -c 5000 -H “Host; service” 192.168.2.211
This is ApacheBench, Version 2.3 <Revision: 655654 >
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation,
Benchmarking 192.168.2.211 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
apr_socket_recv: Connection reset by peer (104)
Total of 2926 requests completed
Against web server directly:
[root@haproxy-A01 ~]# ab -n 5000 -c 5000 -H “Host: service.domain.com” 192.168.2.167
This is ApacheBench, Version 2.3 <Revision: 655654 >
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation,
Benchmarking 192.168.2.167 (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests
Server Software: Microsoft-HTTPAPI/2.0
Server Hostname: 192.168.2.167
Server Port: 80
Document Path: /service.svc
Document Length: 315 bytes
Concurrency Level: 5000
Time taken for tests: 0.741 seconds
Complete requests: 5000
Failed requests: 0
Write errors: 0
Non-2xx responses: 5000
Total transferred: 2460000 bytes
HTML transferred: 1575000 bytes
Requests per second: 6749.50 [#/sec] (mean)
Time per request: 740.796 [ms] (mean)
Time per request: 0.148 [ms] (mean, across all concurrent requests)
Transfer rate: 3242.92 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 51 9.2 53 66
Processing: 31 145 147.7 60 676
Waiting: 1 144 148.0 59 676
Total: 58 196 148.7 107 729
Percentage of the requests served within a certain time (ms)
50% 107
66% 164
75% 314
80% 324
90% 354
95% 387
98% 706
99% 721
100% 729 (longest request)
Can someone help me with fine tunning?
Thank you
Caspi
PS: I had to remove links because of new member. So there are things that are not so correct, ab tests, option check are also now bad.
