I’m having this strange issue where HAProxy randomly stops working for a specific URL/IP and I cannot figure out the reason. A restart of the service will resolve the issue but I need to determine a permanent fix for this rather than a workaround cronjob that restarts the service periodically.
I am proxying a request from a server to an API
When HAProxy is working my logs will look like below (URL replaced with ‘site’ and IP changed for confidentiality):
/var/log/haproxy/access.log-20200117.gz:
haproxy[26037]: 10.10.50.50.:42386 [16/Jan/2020:13:39:14.934] site-uat_site.com site-uat_com/site-uat_com 0/153/272 1104 – 1/1/0/1/0 0/0
and then at some random time, possibly after a request hasn’t been sent to this URL for a while I get the below in my logs:
/var/log/haproxy/access.log-20200118.gz:
haproxy[26037]:10.10.50.10:60657 [17/Jan/2020:12:49:26.065] site-uat_site.com site-uat_com/site-uat_com 0/-1/120008 212 sC 0/0/0/0/3 0/0
Note the 212 error code and sC termination code. I’ve tried doing some research into this:
s: the server-side time-out expired first.
C: waiting for CONNECTION to establish on the server. The server might at most have noticed a connection attempt.
I check the log on the backend of the proxy and I don’t see any traffic reaching it.
Any help or ideas would be greatly appreciated, I’m even a little stuck on how to start troubleshooting this.
Config:
frontend site-uat_site_com
bind 10.10.50.50:6443
mode tcp
log global
option tcplog
option dontlognull
timeout client 90s
use_backend site-uat_com
backend site-uat_com
mode http
timeout connect 30s
timeout server 30s
balance roundrobin
http-request set-header Host nonprod-site.com
server site-uat_com nonprod-site.com:443 ssl verify none
It’s not a 212 error code, it’s 212 bytes read from the server, when it aborted.
Your configuration is very confusing I don’t understand what you are trying to achieve.
You have a HTTP (not HTTPS) request reching port 6443 (why 6443 if it isn’t HTTPS), but you don’t use mode http here, and then you go through a backend in http mode with SSL on the server.
Can you clarify what it is that you are expecting haproxy todo?
Right thank you! I had read that it wasn’t a 212 error code but kept hearing that from my team. Changed my subject line accordingly.
We use HAProxy to be able to support TLS1.2 between a very old in house application and the API the traffic is sent out to, since this traffic goes over the Internet. I also didn’t do the initial config so I can’t speak to why they went this direction, but I do know 6443 could very well be 8086 or something to make more sense but wouldn’t change the behaviour.
However, I did restart HAProxy yesterday evening and once again by this morning the traffic has stopped reaching the API. I haven’t confirmed that the traffic isn’t leaving the server but I have confirmed it’s not reaching the API and I know if I restart HAProxy right now this issue will resolve.
Let me read more into the modes, I have no idea why the frontend is tcp, that does seem odd to me too if you’re saying it should probably be http.
Can you provide the complete configuration and the output of haproxy -vv.
I assume the reason for this are missconfigured timeouts.
Thanks for your assistance Lukas. I’ve included the requested info, I left out the unrelated frontend/backend configs.
[root@server loc]# haproxy -vv
HA-Proxy version 1.7.8 2017/07/07
Copyright 2000-2017 Willy Tarreau willy@haproxy.org
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -DTCP_USER_TIMEOUT=18
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
Global/Default Configs:
global
maxconn 100000
stats socket /var/run/haproxy.stat mode 600 level admin
log 127.0.0.1 local2 debug
chroot /var/empty
ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
ssl-default-bind-options no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
ssl-default-server-options no-tls-tickets
tune.ssl.default-dh-param 1024
daemon
defaults
timeout client 30s
timeout connect 5s
timeout server 5s
frontend dfc-uat_d_com
bind 10.10.50.50:6443
mode tcp
log global
option tcplog
option dontlognull
timeout client 90s
use_backend dfc-uat_com
backend dfc-uat_com
mode http
timeout connect 30s
timeout server 30s
balance roundrobin
http-request set-header Host nonprod-dfc.com
#New PsPrint LZ URL
server dfc-uat_com nonprod-dfc.com:443 ssl verify none
Are you sure you have given me the correct configuration?
As per your error log, the connection timed out after 120 seconds in “timeout connect”:
haproxy[26037]:10.10.50.10:60657 [17/Jan/2020:12:49:26.065] site-uat_site.com site-uat_com/site-uat_com 0/-1/120008 212 sC 0/0/0/0/3 0/0
But none of the configuration snippets you provided matches a 120 second timeout.
I suggest to move from tcp to http mode and upgrade the logging method to http. This will give us a better picture in the log when you hit that issue.
So instead of:
mode tcp
log global
option tcplog
you use
mode http
log global
option httplog
Also, when in this situation, please checkout if you have stale sockets with netstat.
I determined that the problem is with the backend hostname:
server dfc-uat_com nonprod-dfc.com:443 ssl verify none
There are two dynamic IPs associated with nonprod-dfc.com and HAProxy requires a reload/restart to update the IP in it’s cache for this hostname.
Could anyone confirm that I could resolve this with:
resolvers dns
nameserver public-0 xx.xx.xx.xx:53
hold valid 1s
frontend http
bind *:8000
default_backend site-backend
backend site-backend
balance leastconn
server site sub.example.com:80 resolvers dns check inter 1000
Ah ok, that makes sense.
I confirm that DNS resolution as you configured should fix the issue, although in this configuration you make a DNS request every second. Consider using 10 or 30 seconds for the hold valid timeout.