After HAproxy is implemented, application time got increased

Help!
1

Hi,
We have recently implemented Haproxy with two backend servers, and now we are seeing some performance issues with application time got increased, so can you please help me how can i eliminate this slow performance.

image
image.png942×380 23.2 KB

2

here is my config
defaults
log global
mode http
option httplog clf
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
option dontlognull
retries 3
timeout connect 20m
timeout client 10m
timeout server 10m
log-format “%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC \ %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r {sslv/%sslc/%[ssl_fc_sni]/%[ssl_fc_is_resumed]} “%[capture.req.method] %[capture.req.hdr(0)] %[capture.req.uri] %[capture.req.ver]””

#Http In - forward to https
frontend FRONTEND
bind *:80
bind *:443 ssl crt /etc/ssl/hello.pem alpn h2,HTTP/1.1,HTTP/1.0
redirect scheme https code 301 if !{ ssl_fc }
option http-server-close
acl host_abc hdr(host) -i host.com
use_backend WEB if host_abc
http-response set-header Strict-Transport-Security max-age=63072000

backend WEB
mode http
option log-health-checks
option forwardfor
balance roundrobin
cookie ASP.Net_SessionId prefix
cookie SERVERID insert indirect nocache
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https
http-request set-header X-Client-IP %[req.hdr_ip(X-Forwarded-For)]
http-request set-header Host abc.com
server web1xxx.xxx.xx.xx:443 check inter 5s fall 1 rise 1 ssl verify none cookie S1
server web2 xxx.xxx.xx.xx:443 check inter 5s fall 1 rise 1 ssl verify none cookie S2

3

Hi,

A few feedback:

  1. on your diagram you said that HAProxy does HTTP with your server, but in the configuration, HAProxy does SSL…
  2. What version of HAProxy are you using? the ouptut of haproxy -vv would be useful
  3. you have enabled “option http-server-close”, is there any good reason for this?
  4. your timeouts are way too long, furthermore the timeout connect
4

  1. ssl offloading is done on HAproxy is sending traffic over 80, config has typo error
    server web1xxx.xxx.xx.xx:80 check inter 5s fall 1 rise 1 ssl verify none cookie S1
    server web2 xxx.xxx.xx.xx:80 check inter 5s fall 1 rise 1 ssl verify none cookie S2

  2. Haproxy -vv
    HA-Proxy version 2.0.3 2019/07/23 - https://haproxy.org/
    Build options :
    TARGET = generic
    CPU = generic
    CC = gcc
    CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -DMAX_READ_POLL_LOOPS=1
    OPTIONS = USE_OPENSSL=1

Feature list : -EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE -NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE -THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE -LIBCRYPT -CRYPT_H -VSYSCALL -GETADDRINFO +OPENSSL -LUA -FUTEX -ACCEPT4 -MY_ACCEPT4 -ZLIB -SLZ -CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS

Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1c 28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c 28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity(“identity”)
Built without PCRE or PCRE2 support (using libc’s regex instead)
Encrypted password support via crypt(3): yes
Built without multi-threading support (USE_THREAD not set).

Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 2 (2 usable), will use poll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
: mode=HTX side=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS

Available services : none

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
3. there is no particular reason, will this option slow network from backend? can u please explain
4. if i reduce the timeout connect to 5s this gives me 504 gateway timeout error

5

This means HAProxy will open and close a new connection for any single HTTP request passing through. This can slow down things yes.

5s is already too long :slight_smile: in modern infra.
You seem to have a big networking issue between HAProxy and the server, there could be various reasons for that.
This phase of the connection should be from 0 to 2 ms and I am large :slight_smile:

Do you have iptables / conntrack enabled?
Can you share some HAProxy log lines? (don’t forget to anonymise them)
You want to start checking your network and see what happens.

6

IPtables /conntrac are not enabled.