HA Proxy being slow

We have HA Proxy setup on 2 servers for forwarding connections to two backend servers that are running Arcos (a tool used to regulate access management to Linux servers)

We have observed that sessions are taking time while connecting through HA proxy. Also few sessions doesn’t connect at all.

Can you let us know how we can troubleshoot this issue? Is there a ready reckoner of steps that need to be tried out in order to diagnose the issue?

Enable logging and post the logs of the slow requests.

Also provide the configuration and output of haproxy -vv if you need help interpreting those logs.

Hi,

PFB the output of the command:

[root@EDEMUMPIMHAP001 ~]# haproxy -vv
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_F REEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

haproxy -vv output is fine. Need that log then.

Hi,

I have used the following lines in my HA proxy configuration file:

log /dev/log local0 info
log /dev/log local1 notice

which means that the logs for HA Proxy will be in /var/log/messages.

Which format do I convert /var/log/messages into to attach it to this post?

Make sure logging is enabled in the appropriate section as well:

Then post the log of the actual slow request. One is enough, no need to post all of it.

Hi,

This is what I managed to find in /var/log/messages.

Feb 8 09:14:08 EDEMUMPIMHAP001 Keepalived_vrrp[16172]: /usr/sbin/pidof haproxy exited with status 1
Feb 8 09:14:08 EDEMUMPIMHAP001 Keepalived_vrrp[16172]: VRRP_Script(chk_haproxy) failed
Feb 8 09:14:10 EDEMUMPIMHAP001 Keepalived_vrrp[16172]: /usr/sbin/pidof haproxy exited with status 1
Feb 8 09:14:12 EDEMUMPIMHAP001 Keepalived_vrrp[16172]: /usr/sbin/pidof haproxy exited with status 1
Feb 8 09:14:14 EDEMUMPIMHAP001 Keepalived_vrrp[16172]: /usr/sbin/pidof haproxy exited with status 1

Hi,

We have also observed random disconnections when using HA Proxy.

Your assistance will be greatly appreciated, Lukas.

No, you should see for every single request a log line along with a number of timers that will help finding the issue.

Please see the following post for details on the logging configuration:

Ok. I will check the logs and post the actual slow request.

Can you let me know what are the common reasons why disconnections are observed in HA Proxy?

Timeouts.

Hi Lukas,

I am sorry but I am unable to see the logs for HA proxy.
I have posted my haproxy.conf file below:

cat /etc/haproxy/haproxy.cfg

global
log /dev/log local0 info
log /dev/log local1 notice
maxconn 30000

defaults
log global
option dontlognull
timeout connect 20000
timeout client 90000
timeout server 48000
maxconn 30000

option forwardfor except 127.0.0.1

frontend PIMproxy
mode tcp
bind 10.250.5.173:422 transparent
option tcplog
timeout connect 200000
timeout client 100000
timeout server 100000
default_backend dynamic_proxy

backend dynamic_proxy
mode tcp

balance roundrobin

option forwardfor

    timeout server 30000
    timeout connect 40000
    server server1 10.250.5.165:22 check weight 1 fall 2 inter 1s rise 2
    server server2 10.250.5.166:22 check weight 1 backup fall 2 inter 1s rise 2

listen status 10.250.5.174:8888
mode http
stats enable
stats uri /hastatus
stats auth admin:123456

and this is the /etc/rsyslog.d/haproxy.conf file

cat /etc/rsyslog.d/haproxy.conf

local0.=info -/var/log/haproxy/haproxy.log
local1.notice -/var/log/haproxy/haproxy-status.log

Please look at the link posted earlier and enable logging properly.

Hi Lukas,

Your reply hints at the possibility that logging has not been configured properly for this set of servers.
Just to let you know, we have configured logging in precisely the same manner for HAProxy for another set of servers, and are able to see the logs for this server.

This is the configuration file for HA Proxy for the working set of servers:

[root@EDEMUMHAPRXY01 ~]# cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0 info
log /dev/log local1 notice

defaults
log global
option dontlognull
timeout connect 10000
timeout client 45000
timeout server 24000
maxconn 50000

frontend firstpoint
mode tcp
bind 10.250.2.127:54321 transparent
option tcplog
timeout connect 100000
timeout client 50000
timeout server 50000
default_backend trackwizz

backend trackwizz
mode tcp

balance roundrobin

    balance leastconn
    option forwardfor
    timeout server 30000
    timeout connect 40000
    server server1 10.250.0.41:54321 check weight 1 fall 2 inter 1s rise 2
    server server2 10.250.0.36:54321 check weight 1 backup fall 2 inter 1s rise 2

frontend secondpoint
mode tcp
bind 10.250.2.127:8011 transparent
option tcplog
timeout connect 100000
timeout client 50000
timeout server 50000
default_backend trackwizz_2

backend trackwizz_2
balance leastconn
option forwardfor
timeout server 30000
timeout connect 40000
server server1 10.250.0.41:8011 check weight 1 fall 2 inter 1s rise 2
server server2 10.250.0.36:8011 check weight 1 fall 2 inter 1s rise 2

frontend thirdpoint
mode tcp
bind 10.250.2.127:54322
option tcplog
timeout connect 100000
timeout client 50000
timeout server 50000
default_backend trackwizz_3

backend trackwizz_3
balance leastconn
option forwardfor
timeout server 30000
timeout connect 40000
server server1 10.250.0.41:54322 check weight 1 fall 2 inter 1s rise 2
server server2 10.250.0.36:54322 check weight 1 fall 2 inter 1s rise 2

frontend fourthpoint
mode tcp
bind 10.250.2.127:54323
option tcplog
timeout connect 100000
timeout client 50000
timeout server 50000
default_backend trackwizz_4

backend trackwizz_4
balance leastconn
option forwardfor
timeout server 30000
timeout connect 40000
server server1 10.250.0.41:54323 check weight 1 fall 2 inter 1s rise 2
server server2 10.250.0.36:54323 check weight 1 fall 2 inter 1s rise 2

listen status 10.250.2.144:8888
mode http
stats enable
stats uri /hastatus
stats auth admin:12345678
[root@EDEMUMHAPRXY01 ~]#

and here is a snippet of logs for the server:

[root@EDEMUMHAPRXY01 ~]# less /var/log/haproxy/haproxy.log
Feb 23 08:21:53 EDEMUMHAPRXY01 haproxy[30747]: Connect from 10.172.0.239:64734 to 10.250.2.144:8888 (status/HTTP)
Feb 23 08:21:53 EDEMUMHAPRXY01 haproxy[30747]: Connect from 10.172.0.239:64735 to 10.250.2.144:8888 (status/HTTP)
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64721 [23/Feb/2019:08:21:35.118] firstpoint trackwizz/server1 1/0/40700 4300 sD 10/5/5/5/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64722 [23/Feb/2019:08:21:35.200] firstpoint trackwizz/server1 1/0/40618 4696 sD 9/4/4/4/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64724 [23/Feb/2019:08:21:35.202] firstpoint trackwizz/server1 1/0/40616 3379 sD 8/3/3/3/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64714 [23/Feb/2019:08:21:26.496] secondpoint trackwizz_2/server2 1/0/49322 5599 sD 7/3/3/1/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64725 [23/Feb/2019:08:21:35.202] firstpoint trackwizz/server1 1/0/40616 4172 sD 6/2/2/2/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64720 [23/Feb/2019:08:21:35.117] firstpoint trackwizz/server1 1/0/40701 100193 sD 5/1/1/1/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64716 [23/Feb/2019:08:21:28.790] secondpoint trackwizz_2/server2 1/0/47028 11597 sD 4/2/2/0/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64704 [23/Feb/2019:08:21:19.975] secondpoint trackwizz_2/server1 1/0/55844 99109 sD 3/1/1/1/0 0/0
Feb 23 08:22:15 EDEMUMHAPRXY01 haproxy[30747]: 10.172.0.239:64715 [23/Feb/2019:08:21:28.789] secondpoint trackwizz_2/server1 1/0/47029 93409 sD 2/0/0/0/0 0/0

Can you let me know why HA Proxy is able to log all connection requests to haproxy.log for this set of servers, but not able to log connection requests for the server whose file has been posted earlier?

Probably your syslog daemon is configured differently.

Hi Lukas,
Thanks for the tip .

I have checked the systems and it appears that the logs are not being generated on a CentOS system.

I will have to check why this is happening.